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Abstract 


Most community college students are referred to developmental education courses to build basic 
skills. These students often struggle in these courses and college more broadly. CUNY Start is a 
prematriculation program for students assessed as having significant remedial needs. CUNY Start 
students delay matriculation for one semester and receive time-intensive instruction in math, read- 
ing, and writing with a prescribed pedagogy delivered by trained teachers. The program aims to 
help students complete remediation and prepare for college-level courses. This article describes 
the results of an experiment at four community colleges (n ~ 3,800). We estimate that over three 
years, including one semester that students spent in the program and two-and-a-half years after 
the program was complete, CUNY Start substantially increased college readiness, slightly in- 
creased credit accumulation, and modestly increased graduation rates (by increasing participa- 
tion in CUNY’s highly effective ASAP). 


Keywords: Community college, educational reform, postsecondary education, program evalua- 
tion, experimental design. 


Contents 


Acknowledgments 
Abstract 
List of Exhibits 


Introduction 


Developmental Education Reforms 

Content and Pedagogy 

The Time to Complete Developmental Courses 

A Shortage and Underutilization of Student Support Services 
Financial Issues 

Referral to Developmental Education 


Developmental Education Reform at CUNY 


The CUNY Start Model 
Eligible Population 
Program Components 
Theory of Change 


Study Setting 


Method 

Recruitment and Random Assignment 
Estimation Model 

Data and Measures of Academic Outcomes 
Subgroups 


Characteristics of the Sample 


Program Implementation and Service Contrast 
Administration, Cost, and Structure 

Math Instructional Approach 

Reading/Writing Instructional Approach 

Student Support 

Hiring and Training 


Program Effects on Academic Outcomes 
Summary 

Enrollment 

Progress Through Developmental Education 


vii 


iii 


Completion of Gateway Courses 
Progress Through College-Level Courses 
Credential Completion 

A Remark on Mechanism 

Subgroup Findings 

CUNY Start’s Cost 


Discussion 
Appendix A 


References 


Vili 


43 


61 


List of Exhibits 


Table 

1 Baseline Characteristics of Evaluation Sample, by Research Group 

2 Highlights of CUNY Start and Standard College Courses and Services 
3 Impact Estimates on Confirmatory Outcomes After Three Years 


Al Characteristics of Incoming, CUNY Start-Eligible, and CUNY Start Applicant 
Populations at Study Colleges 


A2 Baseline Characteristics of Evaluation Sample, by Research Group 
A3 Enrollment, by Semester 

A4 College Readiness, by Semester 

AS Completion of Gateway Courses, by Semester 

A.6 College-Level Credit Accumulation, by Semester 

A.7 Credential Completion, by Semester 


A8 Number of College-Ready Subject Areas (Out of Three) After Three Years, by 
Subgroup 


AI College-Level Credits Earned After Three Years, by Subgroup 
A.10 Completed a Degree or Certificate After Three Years, by Subgroup 
A.11 Participation in Accelerated Study in Associate Programs (ASAP), by 


Semester 
Figure 
1 Enrollment in CUNY Start or Any College, by Semester 
2 Progress Through Developmental Education, by Semester 
3 College Readiness, by Subject and Semester 
4 Completion of Gateway Courses, by Semester 
5 Cumulative College-Level Credits Earned, by Semester 
6 Credential Completion, by Semester 


1X 


Number of College-Ready Subject Areas (Out of Three) After Three Years, by 
Subgroup 


College-Level Credits Earned After Three Years, by Subgroup 
Credential Completion After Three Years, by Subgroup 


Cumulative College-Level Credits Earned After Three Years 


40 
41 
42 
59 


Introduction 


Community colleges play an important role in U.S. higher education. In fall 2019, over 5,000,000 
students were enrolled in public two-year colleges (National Student Clearinghouse Research 
Center, 2019). Most students who enter community college, however, are deemed academically 
underprepared for college-level courses and are referred to noncredit developmental (remedial) 
education courses to build their math, reading, or writing skills. One national study estimated that 
59% of students entering community colleges were referred to developmental-math courses and 
33% were referred to developmental reading courses (Bailey et al., 2010). These students struggle 
in the developmental education courses and in college more broadly. Nationwide, only 13% of 
students who take a developmental education course graduate within three years (U.S. Depart- 
ment of Education, National Center for Data Statistics, n.d.). Graduation rates are lower still 
among triple-remedial students; that is, students referred to developmental education across math, 
reading, and writing (see, for example, Ginder et al., 2015). 


The low success rates among students referred to developmental education has sparked 
interest in developmental education reform. While there is a growing body of experimental re- 
search on the effectiveness of such reform efforts, there remains a dearth of experimental evidence 
on programs that increase long-term outcomes (for example, credits earned or degree attainment 
through three years) for students referred to developmental education (Bailey et al., 2016). Even 
less evidence exists regarding how to help triple-remedial students (Boatman & Long, 2018). 


Hoping to improve the success rates of its least prepared students, the City University of 
New York (CUNY) developed CUNY Start. CUNY Start’s full-time program was designed for 
incoming students assessed as needing remediation in three subject areas: math, reading, and 
writing. CUNY Start lasts one semester, and its short-term goal is to help students fulfill their 
developmental education requirements, while preparing them for college-level courses. Its long- 
term goal is to improve academic outcomes, including graduation rates. Students enrolling in 
CUNY Start’s full-time program defer college matriculation — which starts when students first 
enroll in degree programs — for one semester to engage in time-intensive instruction in math, 
reading, and writing with a prescribed instructional approach. Students in CUNY Start’s part- 
time program defer matriculation for one semester to engage in time-intensive instruction in math 
or in reading and writing. CUNY Start also provides advising, tutoring, and a weekly seminar 
that teaches students skills they need to succeed in college. Students pay only $75 for the program. 


Among college developmental education reforms, CUNY Start is uncommon in that it 
explicitly aims to recruit triple-remedial students. CUNY Start’s services are also uncommon with 
respect to their greater level of time intensity — with 26.5 hours of class time required per week 
for the full-time program and 13.5 hours of class time required per week for the part-time program 
— and their curricular and pedagogical focus, including a detailed, standardized curriculum and 
a semester-long apprenticeship period for instructors new to CUNY Start. Add to this its low cost 
to students and integrated student support services, and the fact that it occurs before students 


matriculate, and CUNY Start is clearly an innovative approach to grappling with the low success 
rates of students referred to developmental education. 


This article presents three-year findings from a randomized controlled trial (RCT) on the 
effectiveness of CUNY Start. The evaluation was designed to answer the following questions: 


1. What is the effect of the offer to participate in CUNY Start on students’ academic 
progress and completion, compared with that of participating in the colleges’ stand- 
ard courses and services (including the colleges’ standard developmental education 
courses)? Do the effects vary among student populations and settings? 


2. How is CUNY Start implemented, to what degree is it implemented with fidelity to 
the CUNY Start model, and to what degree are there differences between the program 
and the colleges’ standard courses and services? 


3. What are the costs associated with CUNY Start; how do the costs compare with the 
costs of the standard courses and services; and is CUNY Start cost effective? 


This article focuses on the three-year effects of CUNY Start on students’ academic pro- 
gress and completion (research question 1). It briefly summarizes the implementation and cost 
findings (research questions 2 and 3). Detailed implementation findings are discussed in Scrive- 
ner et al. (2018), and detailed cost analyses are discussed in Community College Research Center 
(forthcoming). 


Developmental Education Reforms 


There are a variety of student barriers and institutional practices that help explain the low com- 
pletion rates of community college students in general, and those referred to developmental edu- 
cation programs in particular (for overviews, see Baum et al., 2013; Braxton, 2002; Calcagno et 
al., 2008). While it is beyond the scope of this paper to review all the obstacles, we briefly describe 
those that CUNY Start aims to address, noting reforms, many with features related to CUNY 
Start, that attempt to address these barriers. The barriers include: the way developmental educa- 
tion courses are taught (that is, content and pedagogy), the time it takes to complete these courses, 
a shortage and underutilization of student support services, and financial issues. There is also a 
question as to whether developmental courses are of value at all. 


Content and Pedagogy 


Many developmental education classes focus on building discrete skills using decontex- 
tualized content (Grubb & Gabriner, 2013; Perin, 2013). Courses often use lecture-based peda- 
gogy, in which students are recipients of information but have limited active learning opportuni- 
ties (Rutschow & Schneider, 2011). Traditional lecture-based instruction, when compared to 
“active learning” approaches, may inhibit academic performance (Freeman et al., 2014). Further- 
more, these approaches often fail to engage and prepare students for college-level work (Grubb 
& Gabriner, 2013). 


To improve pedagogy and content, many colleges have implemented multiple math path- 
ways (that is, sets of linked courses designed to teach students math content relevant to their de- 
gree requirements and program of study); self-paced learning models; and flipped classrooms (in 
which students are exposed to content outside of class, while most in-class time is devoted to 
activities, projects, and discussions) (Rutschow & Mayer, 2018). By moving toward an instruc- 
tional approach that positions students as active learners who regularly participate in small-group 
and whole-class activities, and teachers serve as facilitators of the learning process rather than 
lecturers, it is expected that students will be more engaged and develop deep and transferable 
knowledge, stronger academic identities, and an increased commitment to college. Past research 
suggests teaching approaches that encourage active learning are more effective (Stigler & 
Hiebert, 2009). Two recent experimental evaluations of interventions done with math pathways 
that change the content and/or pedagogy of developmental math found positive impacts on stu- 
dents’ successful completion of college-level math (Rutschow et al., 2019) and graduation rates 
(Logue, et al., 2019). 


Reducing the Time to Complete Developmental Courses 


Taking multisemester noncredit developmental education course sequences may slow 
down students’ progress toward graduation (Bettinger & Long, 2005; CUNY Task Force on 
Developmental Education, 2016). These courses increase the number of requirements students 
must complete to earn a degree, increasing the time it takes them to earn a degree (Bettinger & 
Long, 2009). 


To reduce the time it takes to complete developmental education requirements, colleges 
are implementing compressed coursework in math, integrated developmental reading and writing 
courses, and corequisite remediation (that is, students take a college-level course concurrently 
with a developmental education course that supports their learning in the main course) (Rutschow 
& Mayer, 2018). Nonexperimental research on offering developmental courses on a compressed 
time frame suggests improved progress through developmental education and increased course 
pass rates, grades, and rates of persistence (Rutschow & Schneider, 2011). A quasi-experimental 
study of the Accelerated Learning Program (ALP) at Community College of Baltimore County 
— a corequisite remediation intervention — found that students who participated in ALP had 
improved rates of courses attempted and completed (Jenkins et al., 2010). Furthermore, evalua- 
tions of community college and high school programs that incorporate significant increases in 
time spent in targeted subject areas provided evidence of the effectiveness of this approach at 
raising student success (Kemple et al., 2005; Jenkins et al, 2010; Nomi & Allensworth, 2009). 


A Shortage and Underutilization of Student Support Services 


Students referred to developmental education often benefit from recetving adequate 
forms of support to address nonacademic challenges and to develop the academic behaviors, met- 
acognitive skills, and academic confidence associated with college success (Scrivener et al., 2012; 
Bailey et al., 2015). However, there is a broad student support challenge at community colleges 
— for example, the National Academic Advising Association estimates that the median caseload 


of an adviser at public two-year colleges is 441 students (Robbins, 2013) — limiting the amount 
of advising students can receive. 


There is a robust experimental literature on various forms of student support services. 
Experimental evidence on advising interventions finds positive, although often modest, effects 
on students’ academic progress with the use of a variety of modes of advising (Avery et al., 2014; 
Barr & Castleman, 2017; Bettinger & Baker, 2014; Carrell & Sacerdote, 2017; Evans et al., 2017; 
Oreopoulos & Petroniyevic, 2016). Other student support strategies with some limited experi- 
mental evidence of efficacy include tutoring (see, for example, Sommo et al., 2014) and student 
success courses (see, for example, Weiss et al., 2011). There also is experimental evidence that 
learning communities, in which students in two or more classes form cohorts, can have a modest 
positive effect on student outcomes (Richburg-Hayes et al., 2008; Visher et al., 2012; Weiss et 
al., 2015). 


Financial Issues 


Setting aside developmental education, the cost of attending college is a challenge for 
many students (Deming & Dynarski, 2009). Taking noncredit course sequences leads students to 
incur additional tuition costs, and in the process, expend a portion of their financial aid (CUNY 
Task Force on Developmental Education, 2016). Furthermore, taking (and often repeating) non- 
credit course sequences can put students at risk of failing to meet satisfactory academic progress 
requirements for maintaining federal aid eligibility (Scott-Clayton and Schudde, 2017; CUNY, 
2018; U.S. Department of Education, Office of Federal Student Aid, “Staying Eligible,” n.d.-a). 
Federal aid also has time limits: Students are only eligible to receive the equivalent of six years 
of Pell Grant funding (U.S. Department of Education, Office of Federal Student Aid, “Calcu- 
lating Pell Grant Lifetime Eligibility Used,” n.d.-b). The additional time and aid required to 
complete developmental education requirements may inhibit academic progress. Numerous ex- 
perimental evaluations provide causal evidence that finance-related forms can improve student 
academic progress, although often modestly (Angrist, et al., 2014; Angrist et al., 2010; Bettinger 
et al., 2012; Cohodes & Goodman, 2014; Deming & Dynarski, 2009; Goldrick-Rab et al., 2016; 
Mayer et al., 2015). 


Broadly speaking, many developmental education reform strategies have yielded im- 
proved short-term outcomes for students (see, for example, Adelman, 2004; Attewell et al., 2006; 
Bailey et al., 2016; Bailey et al., 2010; Jenkins et al., 2009; Rutschow & Schneider, 2011). Yet, 
when the U.S. Department of Education’s What Works Clearinghouse sought to create a practice 
guide for serving students referred to developmental education, only six strategies were recom- 
mended — none with a “strong” evidence base, three with a “moderate” evidence base, and three 
with a “minimal” evidence base (Bailey et al., 2016). There is a need for more rigorous evidence 
on the effects of programs designed to improve the outcomes of students referred to developmen- 
tal education. 


Referral to Developmental Education 


Developmental education courses themselves may be an impediment to academic pro- 
gress. Valentine et al. (2017) conducted a systematic review of the effects of referral to develop- 
mental education compared with referral to college-level courses. They meta-analyzed data from 
11 regression discontinuity design (RD) studies at two- and four-year institutions. On average, 
they found that referral to developmental education had a negative effect on (a) students’ proba- 
bility of passing the college-level course in which remediation was “needed,” (b) college-level 
credits earned, and (c) degree attainment. Strictly interpreted, these RD studies apply to students 
who were “on the margin” of college readiness; however, it is likely that results apply to students 
falling within at least 0.50 of the cut score (Bloom et al., 2020). Moreover, very recent evidence 
from an experimental evaluation of a multiple measures placement system further suggested that 
referral to developmental education negatively affects students’ academic progress (Barnett, et 
al., 2020). 


The combined weight of the evidence indicates that for the populations where causal ev- 
idence exists, on average, referral to developmental education negatively affects students’ aca- 
demic progress. For students scoring well below placement test cut scores who do not have high 
school GPAs suggesting college readiness, much less is known about the relative effectiveness 
of referral to developmental versus college-level coursework. 


Developmental Education Reform at CUNY 


About 80% of first-time CUNY associate degree-seeking freshmen are assessed as needing re- 
mediation (CUNY Office of Academic Affairs, 2015). During this study, this assessment was 
based primarily on students’ scores on the CUNY Assessment Tests in math, reading, and writing. 
(See the “Data and Measures of Academic Outcomes” subsection for more information on how 
CUNY assessed students’ skills.) At CUNY’s community colleges, the three-year graduation rate 
for first-time, full-time students referred to remediation in at least one subject was 14.2%, which 
applies to students who began school in fall 2011. This rate drops to just 6.5% for triple-remedial 
students (MDRC calculations based on data from CUNY; CUNY Office of Institutional Research 
and Assessment, 2015). 


As is the case at most community colleges, developmental education at CUNY has typi- 
cally comprised multilevel, multisemester noncredit course sequences in math, reading, and writ- 
ing. Students are required to take up to three semesters of developmental education courses. Most 
math courses are taught using lecture-based direct instruction with little emphasis on academic 
skill building (for example, studying and note taking), and reading and writing are typically taught 
separately (see Table 2 for more information). 


As reforms have been explored nationwide, CUNY’s leaders have also focused on re- 
thinking the university system’s approaches, and CUNY and its colleges have implemented many 
different changes. For example, during the period of this study, CUNY was working to revamp 
its developmental education, introducing alterations in how students are assessed and referred to 
courses, as using a single assessment has been critiqued (see, for example, Scott-Clayton, 2012); 


in how students qualify to move out of developmental education; and in the content and pedagogy 
of the courses themselves. CUNY Start is part of these reform efforts as a prematriculation inter- 
vention. It is unique in its targeting of triple-remedial students and students assessed as having 
significant developmental education needs. ! 


The CUNY Start Model 


CUNY Start began in 2009 as a small, intensive program that targeted students with GED certif- 
icates (now, high school equivalency diplomas) who had been assessed as having very weak 
math, reading, and writing skills. It expanded in 2010 to also serve high school graduates. 


Eligible Population 


Students whose placement test scores result in their referral to remediation in math, read- 
ing, or writing are eligible for CUNY Start.* CUNY Start offers a full- and a part-time program. 
The full-time program is only open to students referred to remediation in math and at least one 
other subject area (reading, writing, or both). The part-time program, which provides instruction 
in math or reading and writing, is open to students referred to remediation in one or more subjects. 
Although any student who is referred to remediation is eligible for CUNY Start (and was eligible 
for the evaluation), CUNY Start focuses on recruiting triple-remedial students. Colleges also pri- 
oritize recruiting students referred to two developmental math courses. 


Program Components 


Administration, cost, and structure. At the time of the study, the CUNY Start programs 
were situated in each college’s continuing education division, which offers a range of courses 
outside the academic departments. Administrators in CUNY’s central Office of Academic Affairs 
(OAA) manage the program, working closely with CUNY Start directors at each college. Profes- 
sional development staff based within CUNY OAA provide training to CUNY Start instructors 
and advisers and serve as primary curriculum developers. 


CUNY Start students pay only $75 for the semester, which includes the cost of all course 
materials. Because the program is offered prior to matriculation, students preserve their financial 
aid for future semesters. 


CUNY Start uses a cohort model in which students coenroll in math, reading, and writing 
courses, and a student success seminar, like some learning communities programs. Such grouping 
is hypothesized to foster stronger connections among students and between students and instruc- 
tors (Visher et al., 2008). 


CUNY Start’s program is time intensive, enabling students to complete remedial require- 
ments in a single semester. The full-time program provides up to 26.5 hours of instruction per 


'For more information, see CUNY Task Force on Developmental Education (2016). 
The program also admits a small number of students with limited college experience. 


week during its one semester: 12 hours of math, 12 hours of integrated reading and writing, and 
1.25 to 2.5 hours in the college success seminar.* CUNY Start’s part-time program provides 12 
hours of instruction in math or reading and writing, and 1.5 hours in the seminar. Students do not 
take college-level courses while in CUNY Start. This instructional intensity enables a dramatic 
increase in the amount of time that students spend actively engaged with learning activities such 
as problem-solving. 


Math instructional approach. The core of CUNY Start is time-intensive instruction in 
math and reading/writing using standardized curriculum and pedagogy, which were created by 
experienced faculty members and CUNY Start professional development staff members. Curric- 
ula are regularly refined by the CUNY Start professional development team based on instructor 
feedback. The math curriculum integrates arithmetic and algebra. Various research on math in- 
struction with adults and older adolescents (U.S. Department of Education, 2011) and national 
college-readiness math standards (National Council of Teachers of Mathematics, 2000; National 
Govemors Association Center for Best Practices, 2010; National Mathematics Advisory Panel, 
2008) recommends that math instruction focus on developing students’ math proficiencies (for 
example, problem-solving and adaptive reasoning) and conceptual understanding (for example, 
comprehension of math concepts, operations, and relations). The CUNY Start math curriculum 
models these tenets by emphasizing conceptual understanding, real-world learning, and building 
academic skills such as studying and note taking. 


CUNY Start’s math pedagogy is designed to give students a deep understanding of math 
concepts through student-centered instruction, requiring instructors to limit their use of lectures 
and, instead, promote student discussion. Rather than the traditional remedial pedagogy, charac- 
terized by rote procedural learning and lecture, CUNY Start’s math pedagogy positions students 
as active participants in their own learning, gives them time to think about and struggle with the 
concepts, and encourages them to speak and respond to each other. Central to the pedagogy is the 
technique of questioning, in which instructors ask specific open-ended questions to stimulate stu- 
dent thinking and discussion. Such features are intended to increase the relevancy of subject mat- 
ter in service of deeper understanding and knowledge transfer (Perin, 2013), which have been 
shown to have positive effects on student learning (Hiebert et al., 2003; Stein et al., 2009). 


Reading/writing instructional approach. CUNY Start teaches reading and writing in- 
tegrated in one class to reinforce the relationship between the two and to allow students to move 
more quickly through their developmental education requirements. To engage students, the cur- 
riculum uses a range of culturally relevant texts, including nonfiction articles and essays in the 
beginning of the semester, and short stories and longer works of fiction later in the semester. 


Like CUNY Start math instructors, CUNY Start reading/writing instructors rely on stu- 
dent-centered instruction. Research on reading and writing instruction suggests that students 
should receive explicit instruction in reading comprehension and writing strategies, which are 


At most colleges, the college success seminar for full-time students lasts for 2.5 hours for the first four 
sessions, and then 1.25 hours for the remaining sessions; however, some campuses offer a consistent 1.25- to 
1.5-hour seminar for all full-time students. 


modeled and scaffolded by instructors with a gradual release of responsibility to students through 
repeated practice (National Reading Panel, 2000; Graham & Hebert, 2010; Graham & Perin, 
2007a, 2007b). These practices are further underscored by K-12 national language arts standards, 
which emphasize a development of students’ conceptual understanding, critical thinking skills, 
and ability to apply learned knowledge in real-world settings (National Governors Association 
Center for Best Practices, 2010). 


To emphasize these techniques, CUNY Start reading/writing instructors specifically em- 
ploy the cognitive apprenticeship model, in which instructors aim to help students learn the habits 
and techniques of proficient readers and writers (Collins et al., 1988). For example, instructors 
sometimes “think aloud” to model the processes used by proficient readers and writers — such 
as the process of identifying evidence to support a claim or theme, to identify the author’s point 
of view, or to identify how specific parts of a story are connected to others. 


Student support. CUNY Start’s student support is integrated into the classroom. The 
mandatory college success seminar, led by a CUNY Start adviser, aims to help students develop 
skills to balance school and life, solve problems, advocate for themselves, and view themselves 
as learners. CUNY Start advisers, who are responsible for far fewer students (about 75 students) 
than non-CUNY Start advisers (about 600 students), also meet with students outside the seminar 
to give them support during the program and to plan for their matriculation. CUNY Start math 
tutors and writing assistants provide help to students inside and outside of class. 


Instructor hiring and training. CUNY Start hires instructors based on their content and 
pedagogical knowledge, and their openness to the prescribed curriculum and pedagogy. Once 
hired, CUNY Start instructors participate in a semester of apprenticeship and training before they 
teach their own CUNY Start classes. During this period, they participate in periodic training lead 
by CUNY Start professional developers and are assigned to the classroom of an experienced 
CUNY Start instructor on a full-time basis to observe and eventually assist with instruction in 
order to learn about the CUNY Start curriculum and pedagogy. After that semester, all instructors 
continue to receive ongoing training, classroom observation, and feedback from CUNY Start pro- 
fessional developers.* A growing body of evidence on faculty development in higher education 
suggests that models that embed professional learning opportunities into instructor’s work port- 
folios and are tied closely to everyday practice can increase faculty buy-in (Bickerstaff & 
Edgecombe, 2012; Bragg & Barnett, 2009; Edgecombe et al., 2013). 


Theory of Change 


CUNY Start’s components are designed to work together to improve students’ outcomes. 
The math and reading/writing curricula and pedagogies are central to the model; the instructional 
intensity, instructor training and professional development, and student services are critical to 
facilitating and supporting the instruction. (See Bickerstaff & Edgecombe [2019] for a discussion 
of how the program components interact.) 


4See Kim et al. (in press) for more information on the program model. 


CUNY Start’s underlying theory of change posits that students with substantial develop- 
mental education requirements are best served through a time-intensive, cohort-based model, in 
which a group of students who join the program at the same time take all of their classes together. 
It is designed to build academic preparedness and college skills before matriculation. The pro- 
gram is intended to make students more engaged in their coursework, help them view themselves 
as learners, help them make greater connections with their peers, give them the support they need 
to succeed, and prepare them for college-level work. The program’s low cost to students is ex- 
pected to make it easier for them to participate, and the fact that the program is provided before 
students’ matriculate allows them to preserve their financial aid for future courses. 


The program’s designers hypothesize that, compared with students in standard college 
courses, a higher proportion of CUNY Start students will build their basic skills and complete 
any required developmental education, and that they will do so more quickly. Because CUNY 
Start students spend a semester building their basic skills before matriculating, they are expected 
to earn fewer college credits in the short term. The hypothesis, however, is that if greater amounts 
of CUNY Start students complete their developmental education and become college ready more 
quickly, over the longer term, they will remain enrolled at higher rates, accumulate more college- 
level credits, and graduate more frequently. 


Study Setting 


The CUNY system consists of 25 institutions, including community colleges, four-year colleges, 
and graduate and professional institutions; it serves over a quarter-of-a-million matriculating stu- 
dents each semester (CUNY, n.d.-a). In spring 2015, when the study began, CUNY Start was 
operating at seven CUNY colleges. (At the time this paper was written, it was operating at ten.) 
Of those seven, the evaluation took place at four: Borough of Manhattan Community College, 
Kingsborough Community College, LaGuardia Community College, and Queensborough Com- 
munity College. These colleges were chosen because each had operated the program for at least 
two years, had sufficient program infrastructure to continue operating it, and was willing and able 
to participate in a random assignment evaluation. These are also the four largest CUNY commu- 
nity colleges by total enrollment. 


Borough of Manhattan Community College is in lower Manhattan. It served about 29,000 
students when the study started, including students in credit programs leading to a degree and 
students in continuing education programs. Kingsborough Community College is in Brooklyn. It 
served about 25,000 students at the start of the study. LaGuardia and Queensborough community 
colleges are both located in Queens and served about 36,500 and 19,000 students, respectively, 
at the start of the study (CUNY Office of Institutional Research and Assessment e-mail to author, 
January 3, 2018; CUNY Office of Institutional Research and Assessment, 2018). At Borough of 
Manhattan and Queensborough community colleges, most students take full-time course loads, 
whereas about half of Kingsborough and LaGuardia students do. All four colleges offer a wide 
range of associate’s degree programs that prepare students to transfer to four-year colleges or 


enter professional careers. As mentioned previously, at the time of the study CUNY Start was 
housed in each college’s continuing education division. 


Method 


Recruitment and Random Assignment 


CUNY Start has a robust set of outreach strategies for recruiting students after they have 
applied and been accepted to CUNY. Before each semester, CUNY Start staff members contact 
eligible students by e-mail, mail, and phone, and attend college orientation and registration events 
to describe the program and, during this study, the evaluation. Interested students attend CUNY 
Start information sessions at the colleges to learn more. Students then participate in a structured 
intake process with CUNY Start staff members to help them understand the program’s structure 
and time demands. The goal is for students to make an informed decision about participation 
(Kim et al., in press). 


During the evaluation, students who agreed to participate in CUNY Start and the study 
signed an informed consent form and were then randomly assigned to either the program group 
or control group in real time via a web portal controlled by the research team. Program group 
students were offered the opportunity to participate in CUNY Start. Control group students were 
offered the opportunity to participate in all their colleges’ other programs and services, just not 
CUNY Start (see Table 2 for more information on the standard college courses and services). 
Three cohorts of students are part of the study, one prior to the spring 2015, fall 2015, and spring 
2016 semesters. Immediately prior to random assignment, all students indicated whether they 
intended to participate in CUNY Start’s part- or full-time program if they were assigned to the 
program group (this does not reflect whether they intended to enroll part- or full-time if assigned 
to the control group). 


The analytic sample includes 3,835 individuals (see detailed discussion of the analytic 
sample in the “Characteristics of the Sample” section). The probability of being assigned to the 
program group was 78%, on average. This uneven random assignment ratio was, by design, cho- 
sen to minimize the burden on program staff (who had to overrecruit) and to increase the likeli- 
hood that all program slots would be filled, while ensuring the research goals could be met (for 
example, achieving adequate statistical power). Also, by design, the probability of being assigned 
to the program group was slightly greater in the spring than in the fall.° Around 81% of program 
group members participated in CUNY Start and less than 1% of the control group participated. 
We focus analyses on estimating the effect of the program offer (that is, intention to treat). If 
someone is interested in the effect of CUNY Start on those who complied with their assignment 


SCUNY Start leadership anticipated that there would be fewer CUNY Start-eligible students in the spring 
than in the fall. However, the program had similar capacity to serve students across semesters. Increasing the 
probability of students being offered a spot in CUNY Start in the spring increased the likelihood that all available 
program slots would be filled. 
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(that is, the complier average causal effect), he or she can take the presented estimated effects and 
divide by 0.80 (Angrist et al., 1996; Bloom, 1984; Imbens & Angrist, 1994). 


Estimation Model 


We use the following estimation model to obtain regression-adjusted estimates of the 
average causal effect of the offer to participate in CUNY Start, among people in the evaluation: 

Y=) aX Rp + Deni X Xo+BXT +8. (1) 

Here, Y represents a target outcome (for example, credits earned); R; is a vector of J 
random assignment block indicators (one for each unique college by cohort by intention to enroll 
part versus full time); X, is a vector of student baseline characteristics, including pretest scores, 
random assignment date, and binary indicators for any missing characteristic; and T is set equal 
to one if a student was assigned to the program, and zero otherwise. Robust (Huber-White) stand- 
ard errors are used in all analyses. Analyses for all academic outcomes at all time points presented 
include all 3,835 students, unless otherwise noted (for example, subgroup analyses). 


Data and Measures of Academic Outcomes 


The main data source for academic outcomes was CUNY’s Institutional Research Data- 
base (IRDB). From the IRDB, we received CUNY Start application data (including demographic 
and other background information);° CUNY Start participation data, skills test data, transcript 
data (including the courses that students enrolled in and their grades); and degree data (including 
credentials earned by students at CUNY institutions). We also received data from the National 
Student Clearinghouse (NSC), which includes enrollment and credentials earned by students at 
any of the postsecondary institutions that report to the NSC. NSC data covers 99% of college 
enrollments across all postsecondary institutions in the United States (National Student 
Clearinghouse, n.d.). IRDB and NSC data were received for all students in the study who ap- 
peared in the relevant data. The data covered, at a minimum, spring 2015 (the program semester 
for the first cohort) through winter 2019 (three years after the third cohort’s program semester). 


Several academic outcome measures were created to evaluate CUNY Start’s effects on 
academic progress and credential completion, as will be described. 


Enrollment. A student was considered enrolled if that student participated in CUNY 
Start or matriculated in a CUNY institution according to IRDB data or matriculated in a non- 
CUNY institution according to NSC data. 


College readiness. Entering students at CUNY’s colleges can demonstrate that they are 
prepared for college-level courses in several ways. They can submit scores from their SAT, ACT, 
or New York Regents’ exams, which are statewide standardized exams in core high school sub- 
jects. Students can also take tests administered by CUNY in math, reading, and writing, known 


Only applications collected prior to random assignment were used in any of the analyses presented herein. 
As such, baseline characteristics are missing for 14% of the evaluation sample. 
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as the CUNY Assessment Tests.’ Finally, individual colleges can grant students exemptions on a 
case-by-case basis to allow them to take college-level courses in each subject area. Students in 
developmental education courses were generally required to pass exit tests to demonstrate their 
proficiency.* In our analysis, students were considered college ready in a given subject area (math, 
reading, or writing) if they passed a placement or exit exam in that subject area, passed the highest 
level course in that subject area’s developmental education course sequence, or passed a college- 
level course (including corequisite courses) in that subject area ata CUNY institution, according 
to IRDB data.° 


Completion of gateway courses. A gateway course is defined as the first college-level 
math or English course a student completes. We focused on CUNY courses that may meet 
CUNY’s general education (Pathways) requirements in math and English.!° A student was con- 
sidered to have completed a gateway course if that student received a D or better in a course at a 
CUNY institution, according to IRDB data. 


Cumulative college-level credits. College-level credits were defined as credits earned at 
a CUNY institution, according to IRDB data, that may be applied to satisfy CUNY’s graduation 
requirements. Credits were considered earned if a student received a D or better in a course, and 
credits were considered attempted if a student remained enrolled in the course through the census 
date. Credits associated with remedial-education courses were not included in college-level cred- 
its because they are not degree applicable. 


TAt the start of the study, the math and reading placement tests used by CUNY were COMPASS tests. In 
October 2016, CUNY began using ACCUPLACER rather than COMPASS. The writing placement test, called 
the CUNY Assessment Test in writing, was created by CUNY. While the reading and math tests are untimed, 
the writing test is limited to 90 minutes. Students who come close to passing the reading and math tests may 
retest once after waiting 10 business days. Students may not retake the writing test; however, students who come 
close to passing the test may appeal their score. 

’For reading and writing, the same tests were used for the placement and exit. For math, the exit test used is 
the CUNY Elementary Algebra Final Exam, created by CUNY. In some cases (for example, particular workshop 
courses), students were not required to pass an exit test to demonstrate their proficiency. 

°Passing the highest level course in a developmental course sequence and/or a college-level course provided 
us evidence that students were deemed college ready by CUNY (per authors’ conversations with CUNY staff 
members). Students who passed such courses were considered college ready in that subject in our analysis even 
if we did not have evidence that they achieved a score on a placement or exit test that qualified them as college 
ready. An alternative definition of college readiness that is limited to test scores only would favor the program 
group and increase the program’s estimated impacts on college readiness at the end of three years from 18.0 to 
23.3 percentage points in math, from 2.9 to 7.4 percentage points in reading, and from 5.5 to 7.5 percentage 
points in writing. 

We were only able to measure some outcomes, like college readiness, at CUNY institutions. Thus, a rea- 
sonable concern is that one research group may have been more likely than the other to transfer outside of CUNY 
and achieve the given outcome, thus biasing the impact estimator. According to NSC data, there was no semester 
in which more than 3.7% of the evaluation sample was enrolled at a non-CUNY institution, and the largest 
difference between the two research groups occurred in the first semester, when 1.4% of program group members 
and 0.4% of control group members were enrolled outside of CUNY. 


‘0Math refers to math and quantitative reasoning, and English refers to English composition. The gateway 
courses students take are not necessarily the ones that they and/or their academic advisers will ultimately select, 
as the student nears graduation, to satisfy the Pathways requirement. 
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Credential completion. A credential was defined as a certificate, associate’s degree, 
bachelor’s degree, or graduate degree earned at a CUNY institution according to IRDB data or at 
a non-CUNY institution according to NSC data. 


Outcomes were calculated for each semester. Borough of Manhattan and Queensborough 
community colleges generally have 14-week main sessions (that is, during the fall and spring) 
and four- to eight-week intersessions (that is, during the winter and summer), while Kingsborough 
and LaGuardia generally have 12-week main sessions and six-week intersessions. For all four 
institutions, we combined each main session with its subsequent intersession to create one semes- 
ter. Thus, there are two semesters per year at each college, with one semester generally beginning 
in late August or early September and another beginning between late January and early March. 


Three key confirmatory outcomes were specified prior to the start of the study. Confirm- 
atory outcomes provide answers to sharply focused research questions that address “the study’s 
pre-specified central hypotheses” (Schochet, 2008, p. 3). Number of college-ready subject areas 
is our confirmatory measure of college readiness. It captures progress and completion of devel- 
opmental education course requirements — key milestones toward degree completion and the 
direct goal of the program. Cumulative college-level credits earned is a confirmatory measure of 
student academic progress. It is a proxy for progress toward a degree because students typically 
must earn at least 60 college-level credits to earn an associate’s degree. Finally, Completion of a 
degree or certificate is a confirmatory outcome. It is the long-term outcome in which decision- 
makers are often most interested. !! 


Subgroups 


Subgroup analyses were conducted to examine whether CUNY Start’s impacts varied 
based on four student characteristics (for additional subgroups, see Appendix). 


Full time versus part time. We estimated effects for students who intended to enroll in 
the full-time program, which included up to 26.5 hours of instruction time per week, and for 
students who intended to enroll in the part-time program, which included 13.5 hours of instruction 
time per week. This difference in intervention experience could lead to differential effectiveness. 
Notably, a student’s intended level of attendance in the CUNY Start program was only weakly 
correlated with their observed enrollment intensity in non-CUNY Start courses.!* Furthermore, 
triple-remedial students were more likely to express an intention to participate in the full-time 
CUNY Start program, so this characteristic is somewhat conflated with the amount of remedial 
need. 


''The study’s preanalysis plan states that the completion outcome would consider transfer to a four-year 
institution with 18 or more college-level credits to be a form of completion. For ease of interpretation, however, 
the outcome that excludes transfer is used in the main text and the outcome that accounts for transfer is shown 
in Table A.7 of the online appendix. Results are similar for both outcomes. 

'2 Among the control group, 68% of students who intended to participate in the full-time CUNY Start pro- 
gram attempted a full-time course load of 12 or more credits in the first semester, while 55% of students who 
intended to participate in the part-time program did so. 
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Triple-remedial versus fewer remedial requirements. Number of college-ready sub- 
ject areas at baseline was examined because it is an indicator of a student’s level of risk of not 
achieving the study’s confirmatory outcomes, an important subgroup to consider in almost any 
study, with equity implications. Moreover, this subgroup analysis was motivated by the policy 
importance of understanding whether programs can make a difference for community college 
students with the greatest assessed developmental education needs, an understudied group. Ad- 
ditionally, past research has found differential program effectiveness based on the number and 
types of remedial referrals (Boatman & Long, 2018; Hodara, 2015). 


Intended college. We examined whether program effects vary by college because it is a 
key step in understanding the replicability of a program and the generalizability of its effects. All 
else being equal, a program with consistently positive effects across sites is more likely to be 
effective in similar contexts beyond the study than a program with large differences in effects 
across sites. Recent evidence across the education spectrum has shown that some interventions’ 
effects can vary substantially across settings (Weiss et al., 2017). 


Race/ethnicity. Lastly, we considered whether effects vary by race and ethnicity. Race 
is of importance because Black and Hispanic students have traditionally been under-represented 
in postsecondary education; they enroll disproportionately in community colleges, relative to 
public universities; and persistent race-based achievement gaps are a policy priority (McFarland 
et al., 2019). 


Characteristics of the Sample 


There were 42,232 incoming students at the study colleges during the study’s program semesters 
(spring 2015, fall 2015, and spring 2016). Among those students, 32,462 were assessed as having 
initial remedial needs or had an unknown remedial status, making them technically eligible for 
CUNY Start. There were 4,434 CUNY Start applicants, 41 of whom requested to be removed 
from the study or did not return a signed informed consent form, leaving 4,393 CUNY Start ap- 
plicants for whom we have data. One meaningful difference between the CUNY Start-eligible 
population and CUNY Start applicants is that the applicants were about 30 percentage points 
more likely to have been triple remedial (and, thus, were more likely to require remediation in 
each of the three subject areas) (see Table A.1 in the online appendix). This is consistent with the 
program’s targeted focus on recruiting students assessed as having the most remedial needs. Oth- 
erwise, the observed characteristics of CUNY Start applicants were similar to the eligible popu- 
lation. 


Among the 4,434 students who applied to participate in CUNY Start, 3,873 were ran- 
domly assigned. The other 561 students (13% of applicants) were “exempted,” meaning they 
were guaranteed a spot in CUNY Start.'? Among the students who were randomly assigned, 38 


Exempted students are not included in any analyses. There were three main reasons for exemption: (1) A 
very small number of students were exempted because they had previously participated in CUNY Start’s part- 
time program in one course and were seeking to participate in the part-time program in the other course. (2) Each 
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(0.8% of program group students and 1.6% of control group students) were removed from the 
analytic sample either because they asked to withdraw from the study or because we did not re- 
ceive their signed informed consent forms.'* The analytic sample consisted of 3,835 students, 
including 2,997 program group students and 838 control group students. 


Characteristics of program and control group members at the time of random assignment 
are shown in Table 1. The differences between the two research groups and the p-values associ- 
ated with those differences are provided. In Table 1, “Gender,” “Race/ethnicity,” “Native lan- 
guage,” “Family education,” “Student risk,” and “Highest degree hoping to attain” were derived 
from CUNY Start application data received prior to random assignment. Additional application 
data that were received by CUNY after random assignment were excluded and treated as missing 
due to the possibility (however unlikely) that their collection was dissimilar across the two re- 
search groups. 


The analytic sample included more women than men and was divided almost equally 
between students aged 19 or younger and students aged 20 or older. Black and Hispanic students 
made up most of the sample, and a substantial proportion of the sample had a native language 
other than English. The proportion of students in the sample who were not the first in their family 
to attend college was almost double the proportion of students who were, and there were more 
traditional students than nontraditional students (defined as those who were 24 or older, worked 
35 or more hours per week, had children, or had not received a high school diploma, and were 
not enrolled in high school at the time of random assignment). The vast majority of students in 
the sample hoped to attain a bachelor’s degree or higher. The sample was split almost evenly 
between students who had developmental education requirements in all three subject areas and 
those with fewer requirements, with more students having such requirements in math than in 
reading and writing. Additionally, fewer students scored near the threshold to be considered col- 
lege ready in math than those who scored near the threshold in reading and writing, indicating 
that students in the sample not only were more likely to have a developmental education require- 
ment in math but also had deeper requirements in math than in the other two subject areas.'> 


college was given discretionary exemptions (around 5% of CUNY Start slots). Discretionary exemptions were 
intended for students who, without a spot in CUNY Start, could not matriculate in college. The main reason this 
could have occurred is a student not being able to afford to matriculate and not being eligible for financial aid 
(that is, Pell Grants or New York State aid). (3) Finally, some students sought to participate in CUNY Start after 
the “traditional” semester began (CUNY Start began a short time later). If denied access to CUNY Start, these 
students would have to wait a semester before starting college. 

'4 Among the 3,021 students who were assigned to the program group, 20 were removed from the evaluation 
sample because we did not receive their signed informed consent forms, and 4 were removed because they re- 
quested to withdraw from the study. Among the 852 students who were assigned to the control group, 13 were 
removed from the evaluation sample due to a lack of consent and one requested to withdraw from the study. 

'SNine percent of sample members who had developmental education requirements in math scored within 
ten points of the college-readiness threshold, while 43% and 53% of analogous students scored within ten and 
eight points of the threshold in reading and writing, respectively. 
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Sixty-one percent of the sample intended to participate in the full-time CUNY Start program. 
Two colleges (A and C) accounted for 72% of the evaluation sample. ' 


There were no meaningful differences between program and control group members on 
any of the baseline characteristics shown in Table 1, nor on any of the baseline characteristics 
provided in Table A.2 in the online appendix. Furthermore, an omnibus F-test showed that stu- 
dents’ baseline characteristics were not jointly predictive of their research group (p =.885), indi- 
cating that program and control group students were not systematically different on observable 
characteristics. Random assignment should also produce research groups that are similar on un- 
observed characteristics, such as learning style and tenacity. 


Program Implementation and Service Contrast 


From spring 2015 to spring 2016, the evaluation team collected information on the implementa- 
tion of CUNY Start and the standard offerings at the four colleges using several data sources, 
including interviews with administrators and instructors, observations of classrooms, and surveys 
of students" and instructors. Overall, CUNY Start was implemented with fidelity to the program 
model, and there was a substantial contrast between the program and the colleges’ standard 
courses and services, including their standard developmental education courses. Using data from 
the implementation research, Table 2 compares components of the program with those of the 
standard college courses and services, and this section briefly summarizes those differences. (De- 
tailed implementation findings are available in Scrivener et al. (2018).) 


Administration, Cost, and Structure 


CUNY Start and standard developmental education courses are situated in different parts 
of the colleges. Students pay far less for CUNY Start as a prematriculation intervention than for 
courses that charge standard tuition and fees. 


CUNY Start provides many more hours of classroom instruction than the standard course 
offerings for students referred to developmental education. As mentioned above, CUNY Start’s 
full-time program provides up to 26.5 hours of instruction per week and the part-time program 
provides 13.5 hours of instruction. In contrast, students who are not in CUNY Start might take 
multiple developmental education courses over multiple semesters, with each course typically 
meeting 3 to 6 hours per week. Students can take some college-level courses at the same time, 
and a full-time student usually receives 12 to 16 hours of instruction per week. 


'6College names are redacted here where they may be connected to student outcomes, as such a connection 
is not necessary to answer the study’s research questions. 

"The survey of students was administered during their first semester in the evaluation. All control group 
members and 67% of program group members were surveyed. Seventy-five percent of surveyed students com- 
pleted the survey (75% of the targeted control group members and 75% of the targeted program group members). 
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Math Instructional Approach 


CUNY Start’s math instructional approach is markedly different from standard develop- 
mental math instruction. Standard developmental math courses typically teach arithmetic and al- 
gebra separately and academic skill-building activities are not prevalent. Instructors in standard 
remedial math classes tend to use more lecturing than CUNY Start math instructors, who rely 
more on questioning and student discussion (for more detail, see Bickerstaff & Edgecombe, 
2019). 


Reading/Writing Instructional Approach 


Standard developmental reading and writing were not integrated in three of the four col- 
leges in the study and the courses did not tend to use the cognitive apprenticeship model. In both 
CUNY Start and standard developmental reading and writing, instructors rely on student-centered 
instruction, so the pedagogical difference between CUNY and non-CUNY Start instruction is less 
substantial in reading and writing than it is in math. 


Student Support 


CUNY Start’s student support is more integrated into the classroom than is typical at 
community colleges, and, overall, program group students received somewhat more support than 
control group members. Typically, non-CUNY Start students are not required to participate in a 
college success seminar. Results from a survey of students during their first semester in the study 
found that 78% of program group students took a seminar, compared with 29% of control group 
students. In the survey, 82% of program group students reported that they had at least one one- 
on-one advising session, compared with 65% of control group students. The median program 
group student reported attending two advising sessions, and the median control group student 
reported attending one session — a small difference. Forty-five percent of program group mem- 
bers reported that they received tutoring, compared with 34% of control group students. The me- 
dian program group student reported meeting with a tutor three times, and the median control 
group student reported meeting with a tutor four times. !* 


Hiring and Training 


Typically, college instructors are hired primarily based on their content knowledge (and, 
of course, their academic credentials). In contrast, as mentioned above, CUNY Start hires instruc- 
tors based on their content and pedagogical knowledge, teaching experience, and openness to the 
prescribed curriculum and pedagogy, and it provides a great deal of training to those hired. Most 
CUNY Start instructors who were surveyed for the study had participated in apprenticeships be- 
fore teaching a course for the first time and almost all reported receiving comments on their in- 
struction. In contrast, most non-CUNY Start developmental education instructors who were in- 
terviewed did not report participating in training before teaching their first course. Many reported 


'8CUNY Start students may not have identified every interaction with an embedded classroom math tutor 
or writing assistant as “tutoring,” and, as a result, their survey responses may underreport their engagement with 
tutoring. 
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receiving some kind of professional development, but they participated in it for fewer hours than 
CUNY Start instructors. 


Program Effects on Academic Outcomes 


This section presents estimates of the effect of the opportunity to participate in CUNY Start on 
students’ academic progress and completion during the three years after they entered the evalua- 
tion, including one semester when CUNY Start services were offered to program group members 
(program semester) and two-and-a-half years after the program was complete (postprogram se- 
mesters). We focus on the students’ progress and completion of developmental education, col- 
lege-level credit accumulation, and degree completion. For detailed tables, see the online appen- 
dix. 


Summary 


The key findings are summarized in Table 3. The first two columns show the regression- 
adjusted average outcomes for the program and control groups (with the standard deviations of 
continuous outcomes in parentheses below the average outcomes). The third column presents the 
difference between the means for the two groups, which is the estimated average effect of being 
offered the opportunity to participate in CUNY Start. The last two columns represent the lower 
and upper bounds of a 90% confidence interval for the estimated effect. 


At the end of three years, CUNY Start increased the number of subject areas in which 
students were college ready by an estimated 0.26 subjects (p < .001) — a 13% increase over the 
control group average of 1.99 subject areas in which students were college ready. This finding is 
consistent with the more easily interpretable result that CUNY Start increased the proportion of 
students who were college ready in all three subject areas by 16 percentage points (p < .001) — 
a large effect. 


The estimated effect of CUNY Start on cumulative college-level credits earned was a 
modest 1.35 credits (p = .140), with a 90% confidence interval ranging from -0.16 to 2.86 credits. 
As intended by design, program group students earned fewer credits in the first semester after 
random assignment (the program semester) and then caught up and surpassed the control group 
during the subsequent two-and-a-half years. 


After three years, 14.5% of program group students had earned a degree or certificate, 
compared with 11.4% of control group students (p = .015). As described later in further detail, 
CUNY Start enabled more program group students to become eligible for and participate in 
CUNY’s Accelerated Study in Associate Programs (ASAP), a separate and highly effective three- 
year program. Most of CUNY Start’s graduation effect is likely due to this increase in ASAP 
participation. We proceed by examining these findings in more detail. 
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Enrollment 


We begin by looking at CUNY Start’s effect on enrollment because it provides important 
context for all subsequent findings. Figure 1 shows enrollment in CUNY Start or any college 
across three years, or six semesters, for students offered CUNY Start (that is, the program group, 
squares) and students not offered CUNY Start (that is, the control group, circles). The estimated 
effect of CUNY Start, in percentage points, is plotted in diamonds with bars above and below 
that represent 90% confidence intervals. 


CUNY Start increased enrollment in the first, second, and third semesters after random 
assignment by 10.6, 6.4, and 5.2 percentage points, respectively (p-values ranging from < .001 to 
.008). The effects on enrollment in the fourth, fifth, and sixth semesters were smaller — 3.1, 1.7, 
and 1.1 percentage points, respectively (p-values ranging from .106 to .571).'° Program group 
students may have enrolled at higher rates than control group students in the program semester 
because they found CUNY Start more appealing than business as usual, as it offered them the 
possibility of making substantial progress through developmental education at a low cost and 
without dipping into their financial aid, or because CUNY Start provided them with resources to 
make registration easier. 


Progress Through Developmental Education 


CUNY Start had a substantial positive impact on students’ progress through developmen- 
tal education. Figure 2 plots the confirmatory outcome Number of college-ready subject areas 
across three years. 


At the end of the first semester (the program semester), program group students were 
college ready in 1.87 subject areas, while control group students were college ready in 1.38 sub- 
ject areas. The estimated impact of .49 subject areas after one semester (p < .001) decreased to 
.26 by the end of three years (p < .001), as control group students narrowed the gap during the 
five postprogram semesters. The downward trend of the effect flattened by the end of three years, 
suggesting that a full catch-up by the control group is not expected. 


Figure 3 shows the percentages of program and control group students who were college 
ready in math, reading, writing, and all three subject areas, plotted across three years. 


The pattern in Figure 2 is again exhibited in Figure 3 for the percentage of students who 
were college ready in each subject area. Among the three subject areas, CUNY Start had the 
largest impact on college readiness in math, with a 32.1 percentage point effect after one semester 
that decreased and leveled off at 18.0 percentage points at the end of three years (p < .001 at both 
time points). Impacts on college readiness in reading and writing were smaller, at 8.0 and 9.4 
percentage points after one semester (p < .001 for both subject areas) and decreasing to 2.9 and 


Table A.3 in the online appendix includes results for matriculation in non-CUNY Start courses. As ex- 
pected, the program had a large negative impact on matriculation in non-CUNY Start courses in the first semes- 
ter. In the second through sixth semesters, the effect of CUNY Start on matriculation in non-CUNY Start courses 
was positive, ranging from 1.0 to 4.9 percentage points (p-values ranging from .012 to .581). 
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5.5 percentage points at the end of three years (p = .046 and p <.001), respectively. As one might 
expect, the impact on full college readiness (that is, college readiness in all three subject areas) 
followed a similar pattern, peaking at 24.9 percentage points after one semester and decreasing 
to 15.9 percentage points at the end of three years (p < .001 at both time points).”° 


Completion of Gateway Courses 


CUNY Start succeeded in reducing or eliminating students’ developmental education re- 
quirements. To have an impact on long-term academic outcomes, however, the program must 
prepare students not only to pass a test or course that deems them college ready but also to succeed 
in college-level courses. Whether program group students were more likely than control group 
students to complete a math or English course that satisfies requirements for graduation — that 
is, a gateway course — is the first milestone at which we can test if the program succeeded in 
preparing students for college-level coursework. Figure 4 shows the percentages of program and 
control group students who completed (1) a math gateway course, (2) an English gateway course, 
and (3) both types of gateway courses. 


By the end of three years, CUNY Start increased the percentage of students who com- 
pleted a gateway course in math and English by 5.2 and 4.4 percentage points, respectively (with 
respective p-values of .004 and .022). In the first semester, when program group students were 
enrolled in CUNY Start and therefore not attempting any college-level courses, control group 
students completed gateway courses in each subject at higher rates. By the end of the second 
semester, however, program group students surpassed their control group counterparts with re- 
spect to gateway course completion in each subject. These trends peaked after the third semester, 
and the positive effects were maintained through three years. 


Recall that CUNY Start’s effect on college readiness in math (18.0 percentage points) 
was substantially larger than its effect on college readiness in reading and writing (2.9 and 5.5 
percentage points, respectively). Given this, it is surprising that its effects on completing math 
and English gateway courses were similar — each around five percentage points — representing 
a narrowing of the gap between the program and control groups in math and its maintenance in 
English. 


The narrowing of the gap in math occurred in two steps. First, CUNY Start’s effect on 
attempting a gateway math course by the end of three years was 10.9 percentage points (not 
shown) — a 7.1 percentage point decrease from the 18 percentage point effect on math college 
readiness. Second, CUNY Start’s effect on completing a math gateway course by the end of three 
years was 5.2 percentage points — a 5.7 percentage point further narrowing of the gap from 10.9 
percentage points. This further narrowing occurred because gateway math course pass rates were 
higher in the control group than in the program group among those students who attempted a 


2°CUNY Start’s impact on full college readiness at the end of three years was 15.9 percentage points, while 
the program decreased the percentage of students who were college ready in two subject areas at the end of three 
years by 9.6 percentage points. These findings suggest that the majority of CUNY Start’s impact on full college 
readiness came from students who would have been college ready in two subject areas in absence of the program. 
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math gateway course (not shown). Comparing pass rates among this subset of students requires 
careful interpretation. It may be that traditional developmental math better prepared students to 
pass gateway math courses, or it may be that the type of students CUNY Start induced to attempt 
gateway math were less likely to pass a math gateway course regardless of the method of devel- 
opmental math instruction they received. In any case, CUNY Start’s large impact on college read- 
iness in math was muted by the fact that control group students narrowed the gap when it came 
to attempting and passing gateway math courses. However, by causing more students to attempt 
a gateway math course, CUNY Start caused more students to complete a gateway math course, 
despite lower pass rates among those who attempted such a course. 


Progress Through College-Level Courses 


CUNY Start intends to prepare students not only to complete gateway courses but to 
complete multiple college-level courses and accumulate credits that lead to a college credential. 
Figure 5 shows the average number of cumulative college-level credits earned at CUNY, a con- 
firmatory outcome of the study, across three years. Figure 5 visually represents CUNY Start’s 
theory of change well. The program’s designers hypothesized that the students assessed as being 
the least prepared for college are best served by delaying matriculation for one semester and in- 
tensively addressing their developmental education requirements. By doing so, CUNY Start stu- 
dents will earn fewer college-level credits in the short term, but then be able to accumulate col- 
lege-level credits more quickly in the long term. A reasonable concern, however, is that it might 
not be possible for CUNY Start students to earn college-level credits quickly enough in the post- 
program semesters to make up for the credits foregone in the program semester. 


Program group students and control group students accumulated college-level credits in 
a pattern consistent with CUNY Start’s theory of change. Program group students, on average, 
earned 1.88 fewer credits than control group students in the first semester after random assign- 
ment (the program semester) (p < .001). In semesters two through five, program group students 
earned more credits than control group students in each semester (p-values ranging from < .001 
to .015), and, in semester six, program and control group students earned a similar number of 
credits (p = .615, not directly shown in figure). This increased pace of credit accumulation meant 
that program group students caught up to control group students by the end of the third semester, 
and had earned more credits than control group students by the end of semesters four, five, and 
six, although these differences are fairly small in magnitude (with respective p-values of .230, 
.106, and .140). The evidence does not support concerns that CUNY Start students are not able 
to catch up after delaying matriculation. However, the estimated effect on college credit accumu- 
lation through three years is small, with a 90% confidence interval that ranges from just below 
zero to a moderate three credits. The plateauing trend in the effect estimate suggests that any 
credit effects have largely been achieved. 


Credential Completion 


A program that does not substantially increase the average number of cumulative college- 
level credits earned at the end of three years may nonetheless shift some students across the 
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thresholds necessary to earn a degree or certificate. Figure 6 shows the percentage of program 
group students and control group students who completed a degree or certificate across three 
years. 


Over the first five semesters, there was no meaningful difference between the percentage 
of program group students and control group students who had completed a degree or certificate. 
However, by the end of the sixth semester, 14.5% of program group students and 11.4% of control 
group students had completed a credential, resulting in a 3.1 percentage point impact (p = .015). 
We received enough degree data to examine one additional main session (that is, a spring or fall 
semester) after the three-year mark. With an additional semester, the estimated impact of CUNY 
Start grew to 4.1 percentage points (19.0% versus 14.9%) (p = .003) (not shown).?! These results 
are notable because single-semester higher education programs rarely produce impacts on grad- 
uation (Bailey et al., 2016). 


It seems surprising that CUNY Start had a positive effect on the likelihood of completing 
a credential within three years but had a small estimated effect on cumulative college-level credits 
eamed at the end of three years. To better understand these two seemingly contradictory findings, 
we examined the distribution of cumulative college-level credits earned at the end of three years, 
by research group (see Figure A.1 in the online appendix). At the end of three years, 14.5% of 
program group students and 11.9% of control group students had earned 60 or more college-level 
credits (p = .034), the number of credits typically required to earn an associate’s degree, closely 
mirroring rates of credential completion. Furthermore, a higher proportion of program group stu- 
dents than control group students earned 50-59 credits within three years (6.9% versus 5.7%) (p 
= .194), which is consistent with the finding presented above that CUNY Start’s effect on com- 
pleting a credential increased by one percentage point in the semester immediately following the 
third year.” 


A Remark on Mechanism 


CUNY’s ASAP is an unusually comprehensive, long-lasting (services are provided for 
up to three years), and highly effective student success initiative available to students who are 
fully college ready or have no more than two outstanding developmental education course re- 
quirements. An RCT evaluation of ASAP found that the program substantially increased three- 
year graduation rates (Weiss et al., 2019). 


-1If we exclude certificates and look exclusively at associate’s degrees, the results are similar — CUNY 
Start increased the percentage of students who completed an associate’s degree within three years by 3.0 per- 
centage points, and that impact grew to 4.1 percentage points with an additional semester of data. These results 
are shown in Table A.6 of the online appendix. 

?2CQne may reasonably be concerned that some students who earned 50-59 credits within three years had 
actually earned a credential and that some of their credits (for example, transfer credits) are missing from the 
data. The results are similar when we limit the sample to only students who had not earned a credential, with 
6.3% of program group students and 5.1% of control group students having earned 50-59 college-level credits at 
the end of three years. 
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Most CUNY Start students are not ineligible for ASAP until their developmental educa- 
tion requirements have been reduced. However, CUNY promotes CUNY Start as a “pathway” to 
ASAP and says that it “ensure[s] that eligible students have the opportunity to transition from 
CUNY Start or Math Start to ASAP seamlessly” (CUNY, n.d.-b, pp. 1-2). Given ASAP’s success 
and its partnership with CUNY Start, it was essential to examine CUNY Start’s effect on ASAP 
participation and to consider how much of CUNY Start’s effects on three-year outcomes is likely 
the direct effects of CUNY Start versus indirect effects resulting from enabling more students to 
participate in ASAP. 


CUNY Start succeeded in substantially increasing ASAP participation.”? While very few 
students in either research group participated in ASAP in the first semester after random assign- 
ment, by the end of the second semester, 23.2% of program group students had participated in 
ASAP, compared with only 5.3% of control groups students who had done so. This 17.9 percent- 
age point difference grew to a 19.0 percentage point difference over the following four semesters 
(p < .001 at both time points) (see Table A.11 in the online appendix). CUNY Start likely in- 
creased ASAP participation for two reasons. First, CUNY Start caused more students to become 
eligible for ASAP by enabling them to meet ASAP’s developmental education eligibility require- 
ments. Second, the close relationship between CUNY Start and ASAP succeeded in creating a 
pipeline, increasing the likelihood that ASAP-eligible students joined the program. 


A random assignment evaluation of CUNY’s ASAP (at three of the four colleges partic- 
ipating in this study) showed that the program had a 14.2 percentage point impact on completing 
a degree or certificate within five semesters (Weiss et al., 2019).4 Assuming that this estimated 
effect applies to the 17.9% of students who were successfully induced by CUNY Start to join 
ASAP in the second semester, we estimate that ASAP participation accounted for roughly 2.5 
percentage points (that is, 80%) of CUNY Start’s 3.1 percentage point effect on completing a 
degree or certificate. Taking a similar approach with cumulative college-level credits, we estimate 
that program group students earned 1.26 more credits than control group students based on ASAP 
participation — close to the total estimated effect of 1.35 credits. It is unlikely, however, that 
ASAP participation explains CUNY Start’s effect on progress through developmental education, 
as that impact emerged in the first semester, before program group students began participating 
in ASAP. The practical implications of ASAP’s likely role in these findings is elaborated upon in 
the discussion session. 


Subgroup Findings 


The above results presented the average estimated effect of CUNY Start for the full sam- 
ple; however, different types of students may have responded differently to the program. The 


3Given that CUNY Start increased college readiness, it is likely that the program increased eligibility for 
ASAP as well. We cannot definitively state that CUNY Start increased ASAP eligibility, however, because our 
data do not allow us to identify which students were eligible for ASAP. 


4We use the five-semester ASAP impacts because most program group students who participated in ASAP 
began doing so in their second semester. Thus, they had participated in ASAP for five semesters at the end of 
three years. 
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results presented below document the estimated effects for subgroups of students and the extent 
to which CUNY Start’s impacts varied for different types, or subgroups, of students. Subgroup 
effects are presented for four student characteristics: (1) intended CUNY Start program,” (2) 
number of college-ready subject areas at random assignment, (3) intended college,*° and (4) 
race/ethnicity. The first three subgroups were specified in a preanalysis plan as confirmatory. The 
fourth (race/ethnicity) is included because of its policy relevance. Impact estimates for additional 
subgroups can be found in the online appendix. 


Figures 7, 8, and 9 show CUNY Start’s respective estimated impacts on the number of 
college-ready subject areas, college-level credits earned, and credential completion after three 
years for each subgroup. The estimated effects are plotted with their 90% confidence intervals. 


As shown in Figure 7, CUNY Start had a positive estimated effect on the number of 
college-ready subject areas after three years for all subgroups. CUNY Start’s effect on college 
readiness was greater for students who were college ready in zero subject areas at baseline (that 
is, triple-remedial students) (p-value for the difference in effects = .030) — the students with the 
most room for improvement — narrowing the equity gap. There is some suggestive evidence that 
the program was more effective for full-timers than part-timers, with respect to college readiness 
(p = .132), and evidence that the effects varied by college (p = .100). 


Figure 8 shows that for most subgroups examined, the estimated effects of CUNY Start 
on college-level credits earned were small, positive, and not statistically significant at conven- 
tional levels, with a few exceptions for subgroups with relatively small sample sizes and thus 
imprecise impact estimates (for example, College B and White students, where the estimated 
effects were negative). While the estimated effects vary substantially — for example, from -0.82 
to 2.86 within “Intended college” and from -1.10 to 2.75 within “Race/ethnicity” — there is not 
clear evidence that the true effects vary between subgroups within any of the student characteris- 
tics. 


Perhaps the one exception is the number of college-ready subject areas at random assign- 
ment, where triple-remedial students may have benefited more from the program than those with 
fewer remedial needs (p-value for the difference in effects = .217). CUNY Start is estimated to 
have increased the number of college-level credits earned by 2.45 credits (p = 0.053) for triple- 
remedial students — a potentially important finding given these students also experienced larger- 
than-average effects in college readiness. Meanwhile, the estimated effect for students who had 
fewer developmental education requirements at baseline was near zero. We are cautious to over- 
interpret this finding, and, as will be shown next, there is no evidence that credential completion 
effects were different based on college readiness at baseline. 


5Because this intention was expressed prior to random assignment, a small percentage (5%) of program 
students who participated in CUNY Start in the first semester did not participate in their intended program. 

6Most students either participated in CUNY Start or enrolled in non-CUNY Start courses at their college 
of random assignment. 
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The estimated effects of CUNY Start on completing a degree or certificate were positive 
for most subgroups examined, as shown in Figure 9. The estimated effects did not vary discerni- 
bly based on any of the four characteristics examined (the p-values for differences in effects 
ranged from .527 to .896). As expected, the outliers were the small racial subgroups (for example, 
White, Asian, and other), for which impacts were imprecisely estimated as shown by the wide 
confidence intervals. 


Overall, the subgroup analyses provide suggestive evidence that CUNY Start was more 
effective for triple-remedial students — an important finding given that there is little research on 
how to help this population succeed in higher education (Boatman & Long, 2018). There is little 
clear evidence of any other effect variation. 


CUNY Start’s Cost 


A detailed cost analysis of CUNY Start was conducted by Community College Research 
Center (forthcoming). Using the ingredients method (Levin et al., 2017) and the budgetary 
method, Community College Research Center estimated the per-student costs to operate CUNY 
Start and compared these costs with the educational investment for control group students. This 
comparison yields the incremental costs of CUNY Start; that is, the extra resources colleges 
needed to implement this reform. Over three years, an additional $2,230 was invested in students 
offered a place in CUNY Start. CUNY Start resulted in an additional $340 devoted to students’ 
developmental education; in $790 spent on additional college-credit taking (which was fully off- 
set by the additional revenue generated from this additional credit taking); and in $1,100 spent 
for increased ASAP participation. Through three years, this increased investment led to, as al- 
ready noted, an estimated 15.9 percentage point increase in full college readiness, 1.35 additional 
college-level credits, and a 3.1 percentage point increase in credential completion. With respect 
to college readiness and degree completion, Community College Research Center found that 
CUNY Start lowered the cost-effectiveness ratios — that is, CUNY Start was more efficient at 
producing outcomes than business as usual. The author aptly pointed out that it is important for 
colleges and society to consider their willingness to invest in each earned outcome. 


Community College Research Center (forthcoming, p. 16) noted that “the funding sys- 
tems for CUNY Start and BAU [Business-As-Usual] are not exactly the same. CUNY Start is 
distinctive in its student pricing: students pay only a nominal fixed fee of $75 to enroll in all 
developmental education coursework in the program semester. For BAU, students pay tuition of 
$200 per credit. Additionally, the CUNY Start program is mostly funded upfront by the Univer- 
sity until such time as the Central Office is able to submit for remmbursement from New York 
State for continuing education programming and recoup University funding provided to the col- 
leges to run their programs.” According to CUNY leadership, CUNY Start generates New York 
State Full Time Equivalent (FTE) reimbursements based on seat time — this funding covers most 
budgetary needs and the university supplements the rest. 
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Discussion 


CUNY Start is an innovative approach to helping students assessed as academically underpre- 
pared for college. As the implementation research found, the program differs in several important 
ways from standard college courses and services, including in its cost to the students; its instruc- 
tional intensity; its math and reading/writing curriculum; its pedagogy, especially in math; its 
instructor hiring and training; and the support systems it offers for students. 


Results from our RCT show that over a three-year follow-up period, the program had a 
substantial positive effect on students’ progress through and completion of developmental edu- 
cation, the outcomes that CUNY Start was designed to affect most directly. At the end of three 
years, CUNY Start increased the proportion of students who completed all their developmental 
education requirements by 16 percentage points. The increase in completing developmental edu- 
cation was particularly large in math, which is striking since developmental math is a barrier that 
prevents many students from earning a degree (see Attewell et al., 2006). CUNY Start also suc- 
cessfully enabled many more students to participate in CUNY ASAP, a comprehensive program 
that markedly boosts graduation rates (Weiss et al., 2019). Finally, CUNY Start had a small pos- 
itive estimated effect on college-level credits earned and graduation rates. Importantly, most of 
CUNY Start’s total effect on college-level credits earned and graduation after six semesters is 
likely attributable to CUNY Start increasing ASAP participation and ASAP participation improv- 
ing these outcomes, rather than the direct effect of CUNY Start alone — the significance of this 
is discussed in more detail below. 


CUNY Start targets students who are assessed as needing remediation in math, reading, 
and writing, and, notably, our analyses found that the program was effective for these triple-re- 
medial students. This is important because, as mentioned above, many other developmental edu- 
cation reforms target students with fewer remedial education requirements and/or are unavailable 
to students with the lowest levels of skill in a subject area. 


With respect to increasing college readiness, the program’s most immediate goal, CUNY 
Start is a promising reform. With respect to increasing graduation rates, the results are more com- 
plex. In both cases, it is important to consider the extent that the results likely apply beyond the 
evaluation sample and study institutions. When considering generalizability, the students, insti- 
tutions, alternative services to which CUNY Start was compared, and context are all relevant — 
program effects may vary based on each of these factors (Weiss et al., 2014). We discuss each 
factor, in turn. 


Students. Students in the study were all interested in participating in CUNY Start. Stu- 
dents pursuing the full-time program could dedicate up to 26.5 hours of time to classes per week 
— far more hours than are spent by matriculated students. Even students interested in the part- 
time program could dedicate 13.5 hours to classes per week. The relatively greater quantity of 
class time may be one of the keys to CUNY Start’s effect on progress through developmental 
education, and yet this same aspect of the program may be a barrier to participation for many 
students. Thus, the program may not be the right fit for all students referred to developmental 
education — and, indeed, it was not intended to be the right fit for all developmental education 
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students at CUNY. Reassuringly, among students who were interested in the program and could 
commit to the required time, our subgroup analyses provide suggestive evidence that CUNY Start 
can work for a variety of different types of students. 


Institutions. The study occurred within the context of four community colleges in the 
CUNY system and the program operates as a consortium program comprised of CUNY OAA 
and partner colleges. The colleges are located in a large urban setting and have a diverse mix of 
students with respect to characteristics, including race/ethnicity, country of origin, and languages 
spoken at home. Additionally, during the period discussed in this report, CUNY required students 
to pass skills tests to exit developmental education, while many other community colleges have 
different policies. It is uncertain whether the institutional and personal barriers to student success 
that exist at CUNY and CUNY Start’s approaches to addressing these barriers are fully applicable 
to other community college contexts. 


Alternative services. Relatedly, as Holland (1986, p. 946) aptly noted, “the effect of a 
cause is always relative to another cause.” In other words, there is no singular effect of CUNY 
Start — it depends on what it is being compared with. In the present study, we examine the ef- 
fectiveness of CUNY Start relative to CUNY’s alternative offerings at the time — some of which 
were described earlier. As is the case in many colleges and college systems, at CUNY, the land- 
scape of developmental education has been and remains in flux. CUNY is modifying how stu- 
dents are referred to developmental education, moving away from primarily relying on a place- 
ment test and toward a system using multiple measures and no placement test. Since this change 
is about who is referred to developmental education and CUNY Start is a program option for 
students referred to developmental education, this reform could easily be complementary, alt- 
hough it may change how many and what types of students are eligible for CUNY Start. In addi- 
tion, corequisite remediation is increasing within CUNY with the plan to move away from zero- 
credit postmatriculation offerings over the next several years, whereas corequisite remediation 
was rarely offered during this study. Consequently, even within CUNY, care must be used when 
interpreting this study’s findings in the presence of new alternatives, which were not in existence 
during the present study (recall that the study’s program semesters were spring 2015, fall 2015, 
and spring 2016). 


Context. The graduation effect mostly resulted from CUNY Start enabling greater par- 
ticipation in the highly effecttve CUNY ASAP. In the CUNY context, providing CUNY Start to 
move students into ASAP is a viable strategy for increasing developmental education completion 
and, eventually, graduation. It is worth noting that in addition to providing an intentional pathway 
to ASAP, CUNY Start also caused more students to become eligible to participate in ASAP, since 
students with significant developmental education course requirements (those who are assessed 
as needing more than two developmental education courses) are ineligible for ASAP. It is worth 
considering how the current CUNY Start-to-ASAP pipeline might compare with modifying the 
ASAP eligibility requirements to allow students with greater remedial education requirements to 
enroll directly n ASAP. CUNY ASAP’s estimated effect on three-year degree completion for 
students referred to one (or fewer) remedial education courses is 22 percentage points. For stu- 
dents referred to two or more remedial education courses, the estimated effect is 16 percentage 
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points (Weiss et al., 2019). Might ASAP’s effects for students referred to remediation in math, 
reading, and writing be greater than the three percentage points estimated here (there would be 
cost implications)? 


What about a college considering CUNY Start that does not have ASAP, or something 
similar, to follow? The evidence indicates that the CUNY Start model is well positioned to yield 
substantial improvements in college readiness in the absence of ASAP or something similar. 
However, the evidence from this study does not indicate that the CUNY Start model will mean- 
ingfully increase college-level credit accumulation or graduation rates in the absence of ASAP or 
something similar. It is rare for a one-semester intervention, alone, to meaningfully increase grad- 
uation rates, and our findings suggest that CUNY Start is no exception. 


Developmental education in general, and developmental math in particular, is considered 
a critical barrier to graduation. In this study, CUNY Start caused 16 percentage points more stu- 
dents to complete all developmental education requirements and 18 percentage points more stu- 
dents to complete math developmental requirements, yet the program’s direct effect on gradua- 
tion rates (that is, absent the increased participation in CUNY ASAP) is near zero. This suggests 
that while developmental education may be a barrier to completion for many students, it is likely 
not the only barrier to completion. Figuring out ways to enable more students to complete devel- 
opmental education is only a first step toward substantially boosting graduation rates. This aligns 
with prior research, mentioned earlier, that found that there are multiple institutional and individ- 
ual factors that help explain low graduation rates at community colleges. 


As is often the case, the present evaluation suggests a few areas ripe for further research. 
It is not known whether specific components of the CUNY Start model made more of a difference 
in students’ outcomes than others, yielding important questions for future research. Increased 
class time spent on the developmental education disciplines in a compressed time frame may help 
students focus their learning and gain the subject skills more quickly, even in the absence of 
CUNY Start’s harder-to-implement components (such as the math pedagogy or the specific cur- 
ricula) — this is worth exploring. At the same time, the interactive and participatory curricular 
and pedagogical reform, particularly in math, is quite different from the direct instruction ap- 
proaches that are common in many developmental education classrooms — would these instruc- 
tional reforms make a difference if used in a less time-intensive intervention? 


Overall, this study’s findings demonstrate that with the right mix of reforms, students 
with a variety of developmental education requirements can make considerable progress through 
remedial education. CUNY Start’s innovative model of delaying college matriculation, and fo- 
cusing intensively on building students’ skills and providing them forms of support, yielded 
meaningful positive effects on their completion of developmental education. Importantly, alt- 
hough the program only lasted for one semester, the effects remained large after three years, and 
delaying matriculation did not set back program group students in terms of college-level credits. 
Furthermore, students in the program made progress in developmental education while preserv- 
ing financial aid for potential use on future courses. CUNY’s resource guide (Kim et al., in press) 
can provide information for colleges considering trying the program or something similar on their 
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own campus. In a broader context of developmental education reform, CUNY Start is an intri- 
guing program model to consider for those focused on increasing college readiness. 
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TABLE 1 


Baseline Characteristics of Evaluation Sample, by Research Group 


Characteristic (%) 

Gender 
Female 
Male 
Missing 

Age 
19 or younger 
20 to 23 
24 or older 

Race/ethnicity* 
Hispanic 
White 
Black 
Asian or Pacific Islander 
Other” 
Missing 

Native language 
English 
Spanish 
Other 
Missing 

Family education 
Is the first person in the family to attend college 
Is not the first person in the family to attend college 
Missing 

Student risk® 
Traditional student 
Nontraditional student 
Missing 

Highest degree hoping to attain 
Some college (without degree) 
Associate's degree 
Bachelor's degree 
Postgraduate or professional degree 


Missing 
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Program 


Group 


48.3 
36.7 
15.0 


47.6 
30.4 
22.0 


35.9 
5.9 
25.3 
9.1 
6.4 
17.5 


46.5 
18.6 
19.6 
15.4 


29.0 
53.6 
17.5 


50.4 
37.9 
11.7 


0.4 
10.3 
45.7 
20.4 
23.2 


Control 
Group 


46.2 
40.0 
13.8 


48.6 
28.9 
22.5 


36.9 
5.7 
27.4 
6.8 
7.0 
16.3 


49.0 
17.9 
18.7 
14.4 


28.8 
53.8 
17.4 


52.0 
37.6 
10.3 


0.3 
10.9 
45.2 
21.6 
22.1 


Difference P-Value 
216 
2.1 
-3,3 
1.2 
691 
-1.0 
15 
-0.5 
278 
-1.0 
0.2 
-2.1 
2.3 
-0.6 
1.1 
.620 
-2.6 
0.7 
0.8 
1.0 
992 
0.2 
-0.3 
0.1 
512 
-1.6 
0.3 
1.3 
844 
0.1 
-0.6 
0.5 
-1.2 
1.1 


(continued) 


(continued) 
Program Control 


Characteristic (%) Group Group Difference P-Value 
Number of subject areas requiring developmental education 777 
Zero (fully college-ready) 0.2 0.4 -0.2 
One 15.2 15.4 -0.2 
Two 33.4 33.2 0.2 
Three 51.2 51.0 0.2 
Required developmental education in the subject area 
Math 94.6 94.2 0.3 .696 
Reading 63.4 64.3 -0.9 637 
Writing 77.7 76.4 1.3 419 
Intended college* 1.000 
College A 41.6 41.6 0.0 
College B 16.5 16.5 0.0 
College C 30.4 30.4 0.0 
College D 11.5 11.5 0.0 
Intended CUNY Start program® 1.000 
Full time 61.4 61.4 0.0 
Part time 38.6 38.6 0.0 
Sample size (total = 3,835) 2,997 838 


Sources. CUNY Start application data, MDRC random assignment data, and test data from CUNY's 
Administrative Data Warehouse. 
Note. Values are weighted to account for random assignment ratios across random assignment blocks. 

Rounding may cause slight discrepancies in sums and differences. 

’Respondents who selected "Hispanic" for their ethnicity and chose another race category are 
included only in the "Hispanic" category. Respondents who did not select "Hispanic" for their ethnicity 
and chose more than one racial category are included in the "Other" category. 

>"Other" includes multiracial, Native American/Alaskan native, and other racial/ethnic categories. 

‘Nontraditional students are defined as those who were 24 or older, who worked 35 or more hours 
per week, who had children, or who had not received a high school diploma and who were not enrolled 
in high school at the time of random assignment. Students are listed as "nontraditional" if they fit any 
of these characteristics. Students are considered to be "missing" in the "nontraditional" category if they 
are missing data on two or more of these characteristics and have no other nontraditional characteristic. 

4Random assignment occurred within blocks by cohort, intended college, and intended program. 


31 


TABLE 2 


Highlights of CUNY Start and Standard College Courses and Services 


Component 


CUNY Start 


Standard College Courses and Services 


Administration, cost, 
and structure 


Administration 


Cost to student 


Structure 


Developmental math 


Curriculum 


Pedagogy 


Developmental 
reading/writing 


Curriculum 


Pedagogy 
Student support 


College success 
seminar 


Advising 


Tutoring 


Instructor hiring and 
training 


Hiring 


Training 


Situated in Continuing Education 


$75; students not eligible for financial aid 


1 semester of developmental math, reading, 
and writing; could not take college-level 
courses that semester; up to 26.5 hours of 
instruction per week in the full-time program 


Arithmetic and algebra integrated; 
emphasizes conceptual understanding; 
assignments include activities that develop 
students’ academic skills 


Mostly student-centered instruction 


Reading/writing integrated; writing 
assignments designed to help students 
process and respond to reading material 


Mostly student-centered instruction 


Mandatory; 78% of surveyed students took a 
seminar 


Student-to-adviser ratio was 75:1; 82% of 
surveyed students had at least one one-on- 
one advising session; median student 
reported 2 sessions 


45% of surveyed students received tutoring; 
median student reported 3 tutoring sessions 


Instructors hired based on content and 
pedagogical knowledge and openness to 
CUNY Start instructional approach 


Most instructors participated in an 
apprenticeship; continuing professional 
development was common and regular 


Situated in the Academic Affairs division 


Full-time tuition $2,400; most students 
eligible for financial aid 


Up to 3 semesters of developmental math, 
reading, and writing; students could take 

some college-level courses; 12-16 hours of 
instruction per week for a full-time student 


Arithmetic and algebra taught separately; 
academic skill-building activities not 
prevalent 


Mostly lecture-based instruction 


Reading/writing typically not integrated; 
writing assignments in upper-level courses 
include research synthesis papers 


Mostly student-centered instruction 


Typically not mandatory; 29% of surveyed 
students took a seminar 


Student-to-adviser ratio was ~600:1; 65% 
of surveyed students had at least one one- 
on-one advising session; median student 
reported 1 session 


34% of surveyed students received 
tutoring; median student reported 4 
tutoring sessions 


Instructors typically hired based on content 
knowledge 


Most instructors not trained up-front; 
continuing professional development was 
common but less regular and intensive 


Sources. Field research data; instructor and student survey data; CUNY. (n.d.-c); discussions with CUNY staff members. 
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TABLE 3 


Impact Estimates on Confirmatory Outcomes After Three Years 


Program Control 90% CI 
Outcome Group Group Difference LB UB 
College-ready subject areas (out of 3) 2.25 1.99 0.26 0.20 0.33 


(1.05) (1.10) 


Cumulative college credits earned 22.37 21.02 1.35 -0.16 2.86 
(25.32) (24.30) 


Degree or certificate completion (%) 14.5 11.4 3.1 1.0 5.1 
Sample size (total = 3,835) 2,997 838 


Sources. MDRC calculations from CUNY Institutional Research Database (IRDB) and National 
Student Clearinghouse (NSC) data. 
Note. Standard deviations of continuous outcomes are in parentheses. 

Rounding may cause slight discrepancies in sums and differences. 

Estimates are adjusted by random assignment blocks and select baseline characteristics. 

Cl = confidence interval; LB = lower bound; UB = upper bound. 
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Figure 1 
Enrollment in CUNY Start or Any College, by Semester 
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Sources. MDRC calculations from CUNY Institutional Research Database (IRDB) and 
National Student Clearinghouse (NSC) data. 


Note. Estimates are adjusted by random assignment blocks and select baseline 
characteristics. 
CI = confidence interval. 
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Figure 2 
Progress Through Developmental Education, by Semester 
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Source. MDRC calculations from CUNY Institutional Research Database (IRDB). 
Note. Estimates are adjusted by random assignment blocks and select baseline 
characteristics. 

CI = confidence interval. 
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Figure 3 
College Readiness, by Subject and Semester 
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Source. MDRC calculations from CUNY Institutional Research Database 
(IRDB). 
Note. Estimates are adjusted by random assignment blocks and select baseline 
characteristics. 

CI = confidence interval. 
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Figure 5 
Cumulative College-Level Credits Earned, by Semester 
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Source. MDRC calculations from CUNY Institutional Research Database (IRDB) 
Note. Estimates are adjusted by random assignment blocks and select baseline 
characteristics. 

CI = confidence interval. 


38 


Figure 6 


Credential Completion, by Semester 
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Sources. MDRC calculations from CUNY Institutional Research Database (IRDB) 
and National Student Clearinghouse (NSC) data. 
Note. Estimates are adjusted by random assignment blocks and select baseline 
characteristics. 

CI = confidence interval. 
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Appendix A 


TABLE A.1 
Characteristics of Incoming, CUNY Start-Eligible, and CUNY Start Applicant Populations 
at Study Colleges 


Incoming CUNY Start- CUNY Start 
Characteristic (%) Students Eligible Students Applicants 
Gender 
Female 54.1 54.1 56.4 
Male 46.0 45.9 43.6 
Age 
19 or younger 46.1 47.9 45.9 
20 to 23 28.8 28.7 30.3 
24 or older 25.2 23.4 23.8 
Race/ethnicity* 
Hispanic 38.2 39.3 43.9 
White 13.6 12.5 6.9 
Black 25.7 27.7 31.4 
Asian or Pacific Islander 14.7 13.1 10.1 
Other” iP 74 7.7 
Native language 
English 60.9 60.9 56.0 
Spanish 15.1 16.4 21.9 
Other 23.9 22.7 22.1 
Number of subject areas requiring developmental education 
Zero (fully college-ready) 25.9 0.0 0.3 
One 39.7 53.6 15.0 
Two 19.9 26.8 34.7 
Three 14.5 19.6 50.0 
Required developmental education in the subject area 
Math 57.8 75.2 94.3 
Reading 21.2 27.5 62.8 
Writing 30.4 39.6 77.3 
College® 
College A 37.6 40.1 39.1 
College B 19.3 18.5 17.3 
College C 24.1 23.4 32.3 
College D 19.0 18.0 11.4 
Sample size 42,232 32,462 4,393 
(continued) 
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(continued) 


Sources. Calculations for the incoming and CUNY Start-eligible populations were made by CUNY and 
provided to the authors. Calculations for CUNY Start applicants were made by the authors using 
CUNY Start application data. 

Note. Incoming students are those who were degree seeking, new (either first-time freshmen or 
advanced-standing transfer students), and enrolled at one of the four study colleges during one of the 
study's three program semesters (spring 2015, fall 2015, or spring 2016). CUNY Start-eligible students 
were incoming students who had an initial remedial need or unknown remedial need status. 

Data for incoming and eligible population students is limited to students who matriculated in a 
CUNY institution, while data for CUNY Start applicants is available regardless of whether students 
ultimately matriculated at CUNY or participated in CUNY Start. 

Forty-one CUNY Start applicants requested to be removed from the study or did not return a signed 
informed consent form and are not represented in the table. 

Missing values are not shown and percentages sum to 100%. 

Rounding may cause slight discrepancies in sums and differences. 

‘Respondents who selected "Hispanic" for their ethnicity and chose another race category are 
included only in the "Hispanic" category. Respondents who did not select "Hispanic" for their ethnicity 
and chose more than one racial category are included in the "Other" category. 

>"Other" includes multiracial, Native American/Alaskan Native, and other racial/ethnic categories. 

‘The table represents where CUNY Start applicants intended to enroll and where incoming and 
eligible students actually enrolled. 
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TABLE A.2 


Baseline Characteristics of the Evaluation Sample, by Research Group 


Characteristic (%) 

Diplomas and degrees earned" 
High school diploma 
High school equivalent 
Occupational or technical certificate 
Other 
None of the above 
Missing 

Date of high school graduation or equivalent 
During the past year 
Between one and two years ago 
Between two and five years ago 
More than five years ago 
Missing 

College attendance 
Has previously attended college 
Has not previously attended college 
Missing 

Household status 
Lives with parents 
Lives away from parents 
Missing 

Expenses 
Parents pay more than half 
Parents do not pay more than half 
Missing 

Marital and household status 
Married, living with spouse 
Married, apart from spouse 
Unmarried, living with partner 
Unmarried, not living with partner 
Missing 

Children under the age of 18 


Has one or more children under the age of 18 


Does not have any children under the age of 18 


Missing 
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Program 


Group 


68.6 
13.5 
2.6 
3.4 
1.4 
14.7 


61.8 
3.7 
3.9 
5.9 

24.6 


7.2 
78.1 
14.7 


65.3 
18.6 
16.1 


32.6 
33.2 
34.2 


4.9 
1.6 
10.7 
50.0 
32.9 


9.5 
74.6 
15.8 


Control 
Group 


69.0 
135 
3.4 
4.1 
15 
13.7 


62.3 
4.2 
3.6 
5.9 

24.0 


8.9 
774 
1337 


64.2 
20.4 
15.4 


31.0 
33.2 
35.7 


4.7 
2.6 
9.0 
50.6 
33.2 


10.1 
74.4 
15.5 


Difference P-Value 


-0.4 
0.0 
-0.7 
-0.7 
-0.1 
0.9 


-0.5 
-0.5 
0.3 
0.0 
0.7 


-1.7 
0.7 
1.0 


1.1 
-1.8 
0.7 


1.5 
0.0 
-1.5 


0.2 
-1.0 
1.7 
-0.6 
-0.3 


-0.6 
0.3 
0.3 


771 
787 
.434 
533 
785 
495 
937 


231 


.480 


634 


.246 


.880 


(continued) 


(continued) 


Characteristic (%) 
Dependents 
Has any children or adults dependent on student 
for financial support 
Does not have any children or adults dependent 
on student for financial support 
Missing 
Employment 
Is currently employed 
Is not currently employed 
Missing 
Among those currently employed? 
Number of hours worked per week 
1-10 
11-20 
21-30 
31-40 
More than 40 
Missing 
Highest diploma or degree earned by the parent with 
the most education 
No high school degree 
High school diploma or equivalent 
Some college (but not a degree) 
College degree (AA, BA, MA, PhD) 
Missing 
Financial aid 
Applied for financial aid 
Did not apply for financial aid 
Missing 
Sample size (total = 3,835) 


Program 


Group 


12.1 


20.8 
23.2 
28.6 

4.3 
13.5 


14.2 
22.3 

9.3 
20.4 
33.9 


65.6 
17.3 
17.1 
2,997 


Control 
Group 


13.1 


Sources. CUNY Start application data and MDRC random assignment data. 
Note. Values are weighted to account for random assignment ratios that vary across random assignment 


blocks. 


Rounding may cause slight discrepancies in sums and differences. 


An omnibus F-test was conducted to see whether students' baseline characteristics were jointly 


Difference P-Value 


-1.0 


0.6 


0.4 


-0.9 
0.0 
0.9 


-0.6 
6.4 
-3.9 
0.6 
0.2 
-2.7 


1.7 
1.6 
-0.6 
-1.5 
-1.2 


-3,3 
2.4 
0.9 


.740 


.770 


.101 


492 


156 


predictive of students' random assignment status. The results are not statistically significant (p = 0.885). 
Distributions do not add to 100 percent because categories are not mutually exclusive. 


‘Percentages calculated only among students employed at baseline (n = 1,852). 
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TABLE A.3 


Enrollment, by Semester 


Program — Control 90% CI 
Outcome Group Group Difference LB UB 
Enrolled in CUNY Start or any college (%) 
Semester | 89.5 78.9 10.6 8.1 13.2 
Semester 2 71.6 65.2 6.4 3.4 9.5 
Semester 3 60.9 54 532. 2.0 8.4 
Semester 4 49.1 46.0 3.1 -0.1 6.3 
Semester 5 45.0 43.3 1.7 -1.4 4.9 
Semester 6 39.8 38.8 1.1 -2.0 4.2 
Semesters enrolled 3.56 3.28 0.28 O15 0.41 


(2:02) (41) 


Enrolled in any college (%) 


Semester 1 19.2 78.4 -59.2 -61.9 -56.5 
Semester 2 67.8 64.8 3.0 -0.1 6.0 
Semester 3 60.5 55.6 4.9 1.7 8.1 
Semester 4 49.1 46.0 3.1 -0.1 6.3 
Semester 5 44.9 43.3 1.6 -1.6 4.7 
Semester 6 39.8 38.8 1.0 -2.1 4.1 
Semesters enrolled 2.81 3.27 -0.46 -0.59 -0.32 

(2.00) (2.11) 

Sample size (total = 3,835) 2,997 838 


Sources. MDRC calculations from CUNY Institutional Research Database 

(IRDB) and National Student Clearinghouse (NSC). 

Note. Standard deviations of continuous outcomes are in parentheses. 
Rounding may cause slight discrepancies in sums and differences. 
Estimates are adjusted by random assignment blocks and select baseline 

characteristics. 

CI = confidence interval; LB = lower bound; UB = upper bound. 
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TABLE A.4 
College-Readiness, by Semester 


Outcome 
Number of college-ready subject areas 
Semester 1 
Semester 2 
Semester 3 
Semester 4 


Semester 5 


Semester 6 


College-ready in all three subject areas (%) 


Semester 1 
Semester 2 
Semester 3 
Semester 4 
Semester 5 


Semester 6 


College-ready in math (%) 
Semester | 
Semester 2 
Semester 3 
Semester 4 
Semester 5 


Semester 6 


College-ready in reading (%) 
Semester | 
Semester 2 
Semester 3 
Semester 4 
Semester 5 


Semester 6 


Program 
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Group 


1.87 
(1.09) 
2.07 
(1.09) 
2.16 
(1.07) 
2.20 
(1.06) 
2.23 
(1.06) 
2.25 
(1.05) 


37.9 
48.9 
54.0 
56.6 
58.3 
59.7 


56.8 
61.4 
63.9 
65.5 
66.6 
67.6 


69.7 
76.1 
78.8 
80.2 
80.9 
81.1 


Control 


Group 


1.38 
(0.99) 
1.68 
(1.05) 
1.83 
(1.06) 
1.91 
(1.08) 
1.96 
(1.09) 
1.99 
(1.10) 


13.0 
25.3 
32.7 
37.9 
41.7 
43.8 


24.7 
34.7 
41.4 
45.1 
48.2 
49.6 


61.6 
71.8 
75.0 
76.7 
77.7 
78.2 


Difference 


90% CI 
LB UB 
0.44 0.55 
0.33 0.45 
0.27 0.39 
0.24 0.36 
0.21 0.34 
0.20 0.33 
22.7 27.2 
20.9 26.3 
18.4 24.2 
15.8 21.7 
13.6 19.7 
12.9 19.0 
29.6 34.6 
23.9 29.5 
19.7. 25.5 
17.5 23.3 
15.4 21.4 
15.0 21.0 
5.5 10.5 
1.8 6.8 
1.3 6.3 
1.0 5.9 
0.8 5.6 
0.5 5.3 
(continued) 


(continued) 


Program Control 
Outcome Group Group 
College-ready in writing (%) 
Semester | 60.9 51.6 
Semester 2 69.4 61.7 
Semester 3 73.1 66.2 
Semester 4 74.7 68.7 
Semester 5 75.8 70.1 
Semester 6 76.6 71.0 
Sample size (total = 3,835) 2,997 838 


Difference 


9.4 
7.8 
6.8 
6.0 
5.6 
5.5 


Source. MDRC calculations from CUNY Institutional Research Database (IRDB). 
Note. Standard deviations of continuous outcomes are in parentheses. 
Rounding may cause slight discrepancies in sums and differences. 


Estimates are adjusted by random assignment blocks and select baseline characteristics. 


Cl = confidence interval; LB = lower bound; UB = upper bound. 
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90% CI 

LB UB 
6.7 12.0 
5.1 10.5 
4.1 9.5 
3.3 8.7 
3.0 8.3 
2.9 8.2 


TABLE A.5 


Completion of Gateway Courses, by Semester 


Program 
Outcome (%) Group 
Completed math gateway course 
Semester | 0.6 
Semester 2 18.9 
Semester 3 28.3 
Semester 4 34.0 
Semester 5 38.0 
Semester 6 40.7 
Completed English gateway course 
Semester | 3.4 
Semester 2 35.4 
Semester 3 44.4 
Semester 4 48.9 
Semester 5 52.1 
Semester 6 53.7 
Completed both gateway courses 
Semester | 0.2 
Semester 2 13.2 
Semester 3 22.9 
Semester 4 29.3 
Semester 5 34.3 
Semester 6 37.2 
Sample size (total = 3,835) 2,997 


Source. MDRC calculations from CUNY Institutional Research Database (IRDB). 


Control 
Group 


2.6 
12.4 
19.1 
25.3 
31.6 
35.5 


13.8 
28.7 
37.4 
42.3 
46.9 
49.3 


1.1 

6.9 
14.1 
20.4 
28.0 
31.8 
838 


Difference 


-2.1 
6.5 
9.2 
8.6 
6.4 
5.2 


-10.4 
6.7 
7.0 
6.6 
5.3 
4.4 


-0.9 
6.3 
8.8 
8.9 
6.3 
Res) 


90% CI 
LB UB 
3.0  -1.1 
4.4 8.6 
6.7 11.7 
5.9 11.4 
3.4 9.3 
2.2 8.3 
-12.3, 0 -8.5 
3.9 9.5 
40 10.0 
35 9.7 
2.1 8.4 
1.2 7.6 
-1.5 -0.3 
4.5 8.0 
65 11.1 
63 11.5 
3.4 9.2 
255 8.4 


Note. A "gateway course" is defined as a course that may meet CUNY's general 


education ("Pathways") requirements in math and English. 
Estimates are adjusted by random assignment block and select baseline 


characteristics. 


Rounding may cause slight discrepancies in sums and differences. 
CI = confidence interval; LB = lower bound; UB = upper bound. 
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TABLE A.6 


College-Level Credit Accumulation, by Semester 


Outcome 
Cumulative credits attempted 
Semester 1 
Semester 2 
Semester 3 
Semester 4 


Semester 5 


Semester 6 


Cumulative credits earned 


Semester 1 


Semester 2 


Semester 3 


Semester 4 


Semester 5 


Semester 6 


Sample size (total = 3,835) 


Program 


Group 


0.81 
(2.71) 
7.81 
(7.41) 
14.57 
(12.83) 
20.35 
(18.20) 
25.69 
(23.53) 
29.96 
(27.78) 


0.56 
(2.16) 
5.43 
(6.42) 
10.22 
(11.39) 
14.68 
(16.48) 
18.98 
(21.47) 
yee | 
(25.32) 
2,997 


Control 
Group 


3.56 
(4.21) 
8.78 
(8.27) 
14.26 
(13.06) 
19.16 
(18.10) 
23.85 
(22.64) 
27.86 
(26.75) 


2.45 
(3.65) 
6.13 
(7.35) 
10.08 
(11.78) 
13.95 
(16.37) 
17.73 
(20.55) 
21.02 
(24.30) 
838 


Difference 


-2.75 


-0.97 


0.31 


-1.88 


-0.70 


0.13 


0.72 


1.25 


90% CI 

LB UB 
-2.97  -2.53 
-1.43 -0.51 
-0.46 1.09 
0.10 2.28 
0.45 3.24 
0.44 3.76 
-2.08  -1.68 
-1.12  -0.28 
-0.57 0.84 
-0.27 1.71 
-0.02 2.52 
-0.16 2.86 


Source. MDRC calculations from CUNY Institutional Research Database (IRDB). 

Note. Standard deviations of continuous outcomes are in parentheses. 
Estimates are adjusted by random assignment blocks and select baseline characteristics. 
Rounding may cause slight discrepancies in sums and differences. 
Cl = confidence interval; LB = lower bound; UB = upper bound. 
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TABLE A.7 
Credential Completion, by Semester 


Completed a degree or certificate or transferred to a four-year college" 


Program — Control 90% CI 
Outcome (%) Group Group __ Difference LB UB 
Completed a degree or certificate 
Semester | 0.0 0.0 0.0 0.0 0.0 
Semester 2 0.1 0.0 0.1 0.0 0.2 
Semester 3 0.3 0.3 0.0 = -0.3 0.4 
Semester 4 1.1 1.6 -0.5 -1.2 0.2 
Semester 5 6.9 6.3 0.6 = =-0.9 22, 
Semester 6 14.5 11.4 3.1 1.0 5.1 
Completed an associate's degree 
Semester | 0.0 0.0 0.0 0.0 0.0 
Semester 2 0.0 0.0 0.0 0.0 0.0 
Semester 3 0.1 0.1 0.0 = -0.2 0.2 
Semester 4 0.9 1.3 -0.5 -1.1 0.2 
Semester 5 6.6 6.0 0.6 = =-0.9 2-1 
Semester 6 14.1 11.0 3.0 1.0 5.1 


Semester | 0.0 0.0 0.0 0.0 0.0 

Semester 2 0.1 0.0 0.1 0.0 0.2 

Semester 3 0.8 0.5 0.3 -0.2 0.8 

Semester 4 2.3 2.3 -0.1 -1.0 0.9 

Semester 5 8.6 Ta 0.8 -0.9 2.6 

Semester 6 16.7 13.9 2.8 0.6 5.1 
Sample size (total = 3,835) 2,997 838 


Sources. MDRC calculations from CUNY Institutional Research Database (IRDB) and 
National Student Clearinghouse (NSC) data. 
Note. Estimates are adjusted by random assignment blocks and select baseline 
characteristics. 

Round may cause slight discrepancies in sums and differences. 

CI = confidence interval; LB = lower bound; UB = upper bound. 

"Transfer to a four-year college was defined as being enrolled in a four-year college and 
having 18 or more cumulative college-level credits. 
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TABLE A.8 


Number of College-Ready Subject Areas (Out of Three) After Three Years, by Subgroup 


Characteristic 
Gender 
Male 


Female 


Sample size 
Age 
19 or younger 


20 to 23 


24 or older 


Sample size 
Prior education 


High school diploma 


High school equivalency 


Sample size 
Cohort 
Fall 


Spring 


Sample size 


Source. MDRC calculations from CUNY Institutional Research Database (IRDB). 


N 


1,434 


1,840 


3,274 


1,837 


1,153 


845 


3,835 


2,641 


497 


3,138 


1,556 


2,279 


3,835 


Group 


2.19 
(1.08) 

731 
(1.03) 


2,555 


2.39 
(0.97) 
219 
(1.12) 
2.14 
(1.08) 


2,997 


2.29 
(1.04) 

2.09 
(1.12) 


2,452 
2.36 
(1.03) 


2.18 
(1.06) 


2,997 


Program Control 


Group Difference 


1.89 
(1.15) 

2.14 
(1.01) 


719 


2.09 
(1.02) 
1.87 
(1.13) 
1.90 
(1.21) 


838 


2.02 
(1.08) 

1.97 
(1.09) 


686 
2.15 
(0.93) 


1.88 
(1.22) 


838 


0.31 


0.17 


0.30 


0.26 


0.24 


0.27 


0.12 


Note. Standard deviations of continuous outcomes are in parentheses. 
Estimates are adjusted by random assignment blocks and select baseline characteristics. 
Rounding may cause slight discrepancies in sums and differences. 
CI = confidence interval; LB = lower bound; UB = upper bound. 
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90% CI 

LB UB 
0.20 0.41 
0.08 0.26 
0.21 0.39 
0.13 0.38 
0.10 0.38 
0.19 0.34 
-0.07 0.31 
0.11 0.30 
0.22 0.38 


P-Value 
Difference 
in Effects 
.102 


.779 


.236 


231 


TABLE A.9 
College-Level Credits Earned After Three Years, by Subgroup 


P-Value 
Program Control 90% CI Difference 
Characteristic N Group Group Difference LB UB _in—~Effects 
Gender .690 
Male 1,434 20.37 19.92 0.45 -1.94 2.85 
(24.31) (24.53) 
Female 1,840 24.46 23.20 1.27 -1.08 3.61 
(26.17) (24.18) 
Sample size 3,274 2,555 719 
Age 387 
19 or younger 1,837 24.52 23.72 0.80 -1.49 3.08 
(25.95) (24.27) 
20 to 23 1,153 20.19 18.55 164 -1.17 4.44 
(24.38) (25.04) 
24 or older 845 21.09 16.98 412 0.86 7.37 
(24.90) (22.67) 
Sample size 3,835 2,997 838 
Prior education 385 
High school diploma 2,641 23.05 22.14 0.91 -0.97 2.78 
(25.39) (24.10) 
High school equivalency 497 19.79 16.43 3.35 -0.88 7.59 
(24.72) (22.97) 
Sample size 3,138 2,452 686 
Cohort 164 
Fall 1,556 24.75 24.99 -0.24 -2.75 2.27 
(26.83) (22.81) 
Spring 2,279 20.75 18.29 246 0.50 4.42 


(24.23) (25.23) 
Sample size 3,835 2,997 838 


Source. MDRC calculations from CUNY Institutional Research Database (IRDB). 

Note. Estimates are adjusted by random assignment blocks and select baseline characteristics. 
Standard deviations of continuous outcomes are in parentheses. 
Rounding may cause slight discrepancies in sums and differences. 
CI = confidence interval; LB = lower bound; UB = upper bound. 
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TABLE A.10 


Completed a Degree or Certificate After Three Years, by Subgroup 


Characteristic (%) 
Gender 
Male 


Female 


Sample size 
Age 
19 or younger 
20 to 23 
24 or older 


Sample size 
Prior education 
High school diploma 


High school equivalency 


Sample size 
Cohort 
Fall 
Spring 


Sample size 


N 
1,434 
1,840 
3,274 
1,837 


1,153 
845 


3,835 
2,641 

497 
3,138 
1,556 
2,279 


3,835 


Program Control 


Group 


11.6 
17.0 


2,555 
16.6 


12.1 
13.5 


2,997 


14.7 
12.7 


2,452 


18.2 
11.9 


2,997 


Group 
9.8 
12.8 
719 
12.0 


11.7 
8.0 


838 


11.9 
5.6 


686 


13.9 
9.6 


838 


Difference 


1.8 
4.2 


4.7 
0.3 
BES) 


2.8 
7.1 


4.3 
2.3 


P-Value 
90% CI Difference 
LB UB in Effects 
413 
-1.4 5.1 
0.8 7.6 
255 
1.4 8.0 
-3.5 4.2 
1.1 10.0 
262 
0.1 5.4 
14 128 
461 
0.6 8.1 
-0.3 4.9 


Source. MDRC calculations from CUNY Institutional Research Database (IRDB) and National 

Student Clearinghouse (NSC) data. 

Note. Estimates are adjusted by random assignment blocks and select baseline characteristics. 
Rounding may cause slight discrepancies in sums and differences. 
Cl = confidence interval; LB = lower bound; UB = upper bound. 
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TABLE A.11 


Participation in Accelerated Study in Associate Programs (ASAP), by Semester 


Outcome (%) 
Ever participated in ASAP 
Semester | 
Semester 2 
Semester 3 
Semester 4 
Semester 5 
Semester 6 
Sample size (total = 3,835) 


Source. MDRC calculations from CUNY Institutional Research Database (IRDB). 


Program 


Group 


0.7 
23.2 
27.0 
28.7 
29.2 
29.8 

2,997 


Control 


Group 


0.9 
5.3 
7.8 
9.0 
10.3 
10.8 
838 


Difference 


-0.2 
17.9 
19.2 
19.7 
19.0 
19.0 


90% CI 
LB 


-0.7 
16.0 
17.1 
1735 
16.7 
16.7 


Note. Estimates are adjusted by random assignment blocks and select baseline 


characteristics. 


Rounding may cause slight discrepancies in sums and differences. 
CI = confidence interval; LB = lower bound; UB = upper bound. 
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UB 


0.4 
19.8 
21.2 
21.9 
21,2 
21.3 


OV 


‘quoo1ed OT = x ‘quaoIed ¢ = x4. {WUddIOd | = yxy ‘SPOADT DOUBDTFIUSIS [ROSES 
“SOlJSLIO}IVIVYO SUT[ISeQ JO9[9S pue SYdOTq JUOTUUSISse WopuRI Aq poysn[pe ose soyeUIs| ‘ajoy 
‘(AAU osequyeq yoreasoy [euoNISU, ANNO Wo suonefnsjes QYCW ‘'22/no¢ 


weisclg@ jonuojo 
(%) ddULI UIYIM S}Iposo JO Jaquinu pouie|y 
Se 0€ SC 0c ST Or iS 0 


8L'0 


61-01 
6C-0C 
6£-0€ 
6b-0F 


6S-0S 


POUIvd S}IPOIO [OAa]-98aT[OD 


69-09 


6L-0L 


+08 


SADAX AY] AAU{P PIUADT SIIPAAD [aAaT-asapJoy aayojnuiny 


Tv oansiy 


59 


References 


Adelman, C. (2004). Principal indicators of student academic histories in postsecondary 
education, 1972-2000. Washington, DC: U.S. Department of Education, Institute of 
Education Sciences. 


Angrist, J., Autor, D., Hudson, S., & Pallais, A. (2014). Leveling up: Early results from a 
randomized evaluation of post-secondary aid (NBER Working Paper Series, No. 20800). 
Cambridge, MA: National Bureau of Economic Research. 


Angrist, J., Oreopoulos, P., & Williams, T. (2010). When opportunity knocks, who answers? 
New evidence on college achievement awards (NBER Working Paper Series, No. 16643). 
Cambridge, MA: National Bureau of Economic Research. 


Angrist, J. D., Imbens, G. W., & Rubin, D. B. (1996). Identification of causal effects using 
instrumental variables. Journal of the American Statistical Association, 91(434), 444-455. 


Attewell, P. A., Lavin, D. E., Domina, T., & Levey, T. (2006). New evidence on college 
remediation. Journal of Higher Education, 77(5), 886-924. 


Avery, C., Howell, J., & Page, L. (2014). A review of the role of college counseling, coaching, 
and mentoring on students’ postsecondary outcomes. New York, NY: College Board. 


Bailey, T., Bashford, J., Boatman, A., Squires, J., Weiss, M., Doyle, W., Valentine, J. C., 
LaSota, R., Polanin, J. R., Spinney, E., Wilson, W., Yeide, M., & Young, S. H. (2016). 
Strategies for postsecondary students in developmental education. Washington, DC: 
Institute of Education Sciences, What Works Clearinghouse. 


Bailey, T., Jeong, D. W., & Cho, S.-W. (2010). Referral, enrollment, and completion in 
developmental education sequences in community colleges. Economics of Education 
Review, 29(2), 255-270. 


Bailey, T. R., Jaggars, S. S., & Jenkins, D. (2015). Redesigning America’s community colleges: 
A clearer path to student success. Cambridge, MA: Harvard University Press. 


Barnett, E., Cullinan, D., Kopko, E., & Belfield, C. (2020). Who should take college-level 
courses? Impact findings from an evaluation of multiple measures assessment 
[Unpublished manuscript]. 


Barr, A., & Castleman, B. (2017). The bottom line on college counseling [Working paper]. 
College Station, TX: Texas A&M University. 


Baum, S., Kurose, C., & McPherson, M. (2013). An overview of American higher education. 
Future of Children, 23(1), 17-39. 


Bettinger, E. P., & Baker, R. B. (2014). The effects of student coaching: An evaluation of a 
randomized experiment in student advising. Educational Evaluation and Policy Analysis, 
36(1), 3-19. 


Bettinger, E. P., & Long, B. T. (2005). Remediation at the community college: Student 
participation and outcomes. New Directions for Community Colleges, Number 129, 17-26. 


61 


Bettinger, E. P., & Long, B. T. (2009). Addressing the needs of underprepared students in 
higher education: Does college remediation work? Journal of Human Resources, 44(3), 
736-771. 


Bettinger, E. P., Long, B. T., Oreopoulos, P., & Sanbonmatsu, L. (2012). The role of 
simplification and information in college decisions: Results from the H&R Block FAFSA 
experiment. Quarterly Journal of Economics, 127(3), 1205-1242. 


Bickerstaff, S., & Edgecombe, N. (2012). Pathways to faculty learning and pedagogical 
improvement. /nside Out, 1(3), 1-4. 


Bickerstaff, S., & Edgecombe, N. (2019). Teaching matters and so does curriculum: How 
CUNY Start reshaped instruction for students referred to developmental mathematics. 
(CCRC Working Paper No. 110). New York, NY: Community College Research Center, 
Teachers College, Columbia University. 


Bloom, H., Bell, A., & Reiman, K. (2020). Using data from randomized trials to assess the 
likely generalizability of educational treatment-effect estimates from regression 
discontinuity designs. Journal of Research on Educational Effectiveness, 1-30. 


Bloom, H. S. (1984). Accounting for no-shows in experimental evaluation designs. Evaluation 
Review, 8(2), 225-246. 


Boatman, A., & Long, B. T. (2018). Does remediation work for all students? How the effects of 
postsecondary remedial and developmental courses vary by level of academic preparation. 
Educational Evaluation and Policy Analysis, 40(1), 29-58. 


Bragg, D. D., & Barnett, E. (2009). Lessons learned from breaking through. /n Brief: 
Champaign, IL: Office of Community College Research and Leadership, College of 
Education, University of Illinois at Urbana-Champaign. 


Braxton, J. M. (2002). Reworking the student departure puzzle. Nashville, TN: Vanderbilt 
University Press. 


Calcagno, J. C., Bailey, T., Jenkins, D., Kienzl, G., & Leinbach, T. (2008). Community college 
student success: What institutional characteristics make a difference? Economics of 
Education Review, 27, 632-645. 


Carrell, S., & Sacerdote, B. (2017). Why do college-going interventions work? American 
Economic Journal: Applied Economics, 93), 124-151. 


Cohodes, S. R., & Goodman, J. S. (2014). Merit aid, college quality, and college completion: 
Massachusetts’ Adams Scholarship as an in-kind subsidy. American Economic Journal: 
Applied Economics, 6(4), 251-285. 


Collins, A., Brown, J. S., & Newman, S. E. (1988). Cognitive apprenticeship: Teaching the 
Craft of reading, writing, and mathematics. Thinking: The Journal of Philosophy for 
Children, 8(1), 2-10. 


Community College Research Center. (Forthcoming). Cost and cost-effectiveness analysis of 
CUNY Start [Manuscript in publication]. 


CUNY. (n.d.-a). About. https://www.cuny.edu/about. 


62 


CUNY. (n.d.-b). CUNY start, math start, ASAP, Accelerated study in associate programs 
[Brochure]. http://www 1.cuny.edu/sites/asap/wp- 
content/uploads/sites/8/2018/10/CS_MS ASAP Brochure.pdf. 


CUNY. (n.d.-c). CUNY Start Program Overview. 
http://www 1.cuny.edu/sites/cunystart/program/cuny-start 


CUNY Office of Academic Affairs. (2015). Remediation at CUNY: Update to the CUE 
Council. https://www.cuny.edu/wp-content/uploads/sites/4/page- 
assets/about/administration/offices/undergraduate- 
studies/coordinated/CUERemediationUpdate 1 2-16-20150US.pdf. 


CUNY Office of Institutional Research and Assessment. (2015). Three-year graduation rates of 
full-time, first-time freshmen enrolled in associate or certificate programs by gender: Fall 
2011 entering cohort. New York, NY: City University of New York. 


CUNY Office of Institutional Research and Assessment. (2018). Total enrollment by 
undergraduate and graduate level, full-time/part-time attendance, and college. New York, 
NY: City University of New York. 


CUNY Task Force on Developmental Education. (2016). Report of the CUNY task force on 
developmental education. https://www?2.cuny.edu/wp-content/uploads/sites/4/page- 
assets/about/administration/offices/undergraduate-studies/developmental- 
education/Proposed-Recommendations-of-RTF-06.17.16.final_.pdf. 


Deming, D., & Dynarski, S. (2009). Into college, out of poverty? Policies to increase the 
postsecondary attainment of the poor (NBER Working Paper No. 15387). Cambridge, 
MA: National Bureau of Economic Research. 


Denning, J. T. (2017). College on the cheap: Consequences of community college tuition 
reductions. American Economic Journal: Economic Policy, 2), 155-188. 


Edgecombe, N., Jaggars, S. S., Baker, E. D., & Bailey, T. (2013). Acceleration through a 
holistic support model: An implementation and outcomes analysis of FastStart@CCD. 
New York, NY: Community College Research Center, Teachers College, Columbia 
University. 


Evans, W. N., Kearney, M. S., Perry, B. C., Sullivan, J. X. (2017). Increasing community 
college completion rates among low-income students: Evidence from a randomized 
controlled trial evaluation of a case management intervention (NBER Working Paper No. 
24150). Cambridge, MA: National Bureau of Economic Research. 


Ginder, S. A., Kelly-Reid, J. E., & Mann, F. B. (2015). Graduation rates for selected cohorts, 
2006-11; student financial aid in postsecondary institutions, academic year 2013-14; and 
admissions in postsecondary institutions, fall 2014, first look (provisional data). 
Washington, DC: National Center for Education Statistics, Institute of Education Sciences, 
U.S. Department of Education. 


Goldrick-Rab, S., Kelchen, R., Harris, D. N., & Benson, J. (2016). Reducing income inequality 
in educational attainment: Experimental evidence on the impact of financial aid on college 
completion. American Journal of Sociology, 121(6), 1762-1817. 


63 


Graham, S., & Hebert, M. (2010). Writing to read: Evidence for how writing can improve 
reading (Carnegie Corporation Time to Act Report). Washington, DC: Alliance for 
Excellent Education. 


Graham, S., & Perin, D. (2007a). What we know, what we still need to know: Teaching 
adolescents to write. Scientific Studies of Reading, 11(4), 313-335. 


Graham, S., & Perin, D. (2007b). Writing next: Effective strategies to improve writing of 
adolescents in middle and high school. Washington, DC: Alliance for Excellent Education. 


Grubb, W. N., & Gabriner, R. (2013). Basic skills education in community colleges: Inside and 
outside of classrooms. New York, NY: Routledge. 


Hiebert, J., Gallimore, R., Garnier, H., Givvin, K. B., Hollingsworth, H., Jacobs, J., Miu-Ying 
Chui, A., Wearne, D., Smith, M., Kersting, N., Manaster, A., Tseng, E., Etterbeek, W., 
Manaster, C., Gonzales, P., & Stigler, J. (2003). Teaching mathematics in seven countries: 
Results from the TIMSS 1999 video study. Washington, DC: National Center for Education 
Statistics, Institute of Education Sciences, U.S. Department of Education. 


Hodara, M. (2015). The effects of English as a second language courses on language minority 
community college students. Educational Evaluation and Policy Analysis, 37(2), 243-270. 


Holland, P. W. (1986). Statistics and causal inference. Journal of the American Statistical 
Association, 81(396), 945-960. 


Imbens, G. W., & Angrist, J. D. (1994). Identification and estimation of local average treatment 
effects. Econometrica, 62(2), 467-475. 


Jenkins, D., Jaggars, S. S., & Roksa, J. (2009). Promoting gatekeeper course success among 
community college students needing remediation: Findings and recommendations from a 
Virginia study [Summary report]. New York, NY: Community College Research Center, 
Teachers College, Columbia University. 


Jenkins, D., Speroni, C., Belfield, C., Jaggars, S. S., & Edgecombe, N. (2010). 4 model for 
accelerating academic success of community college remedial English students: Is the 
Accelerated Learning Program (ALP) effective and affordable? (CCRC Working Paper 
Series). New York, NY: Community College Research Center, Teachers College, 
Columbia University. 


Kemple, J. J., Herlihy, C. M., & Smith, T. J. (2005). Making progress toward graduation: 
Evidence from the talent development high school model. New York, NY: MDRC. 


Kim, J., Simon, M., Horenstein, A., & Webber, A. (in press). CUNY Start: A resource guide for 
reforming developmental education in the college transition space. New York, NY: City 
University of New York. 


Levin, H. M., McEwan, P. J., Belfield, C., Bowden, A. B., & Shand, R. (2017). Economic 
evaluation in education: Cost-effectiveness and benefit-cost analysis. Thousand Oaks, CA: 
SAGE. 


Logue, A. W., Douglas, D., & Watanabe-Rose, M. (2019). Corequisite mathematics 
remediation: Results over time and in different contexts. Educational Evaluation and 
Policy Analysis, 41(3), 294-315. 


64 


Mayer, A. K., Patel, R., Rudd, T., & Ratledge, A. (2015). Designing scholarships to improve 
college success. New York, NY: MDRC. 


McFarland, J., Hussar, B., Zhang, J., Wang, X., Wang, K., Hein, S., Diliberti, M., Forrest 
Cataldi, E., Bullock Mann, F., & Barmer, A. (2019). The condition of education 2019. 
Washington, DC: National Center for Education Statistics. 


National Council of Teachers of Mathematics. (2000). Principles and standards for high school 
mathematics. Reston, VA: National Council of Teachers of Mathematics. 


National Governors Association Center for Best Practices. (2010). Common core state 
standards. Washington, DC: Council of Chief State School Officers. 


National Mathematics Advisory Panel. (2008). Foundations for success: The final report of the 
national mathematics advisory panel. Washington, DC: U.S. Department of Education. 


National Reading Panel. (2000). Teaching children to read: An evidence-based assessment of 
the scientific research literature on reading and its implications for reading instruction. 
Bethesda, MD: National Institute of Child Health and Human Development. 


National Student Clearinghouse. (n.d.). About the Clearinghouse. 
https://www.studentclearinghouse.org/about/. 


National Student Clearinghouse Research Center. (2019). Term enrollment estimates: Fall 
2019. Herndon, VA: National Student Clearinghouse Research Center. 


Nomi, T., & Allensworth, E. (2009). “Double-dose” algebra as an alternative strategy to 
remediation: Effects on students’ academic outcomes. Journal of Research on Educational 
Effectiveness, 2(2), 111-148. 


Oreopoulos, P., & Petronijevic, U. (2016). Student coaching: How far can technology go? 
(NBER Working Paper 22630). Cambridge, MA: National Bureau of Economic Research. 


Perin, D. (2013). Literacy skills among academically underprepared students. Community 
College Review, 41(2), 118-136. 


Richburg-Hayes, L., Visher, M. G., & Bloom, D. (2008). Do learning communities effect 
academic outcomes? Evidence from an experiment in a community college. Journal of 
Research on Educational Effectiveness, 1(1), 33-65. 


Robbins, R. (2013). Implications of advising load. In A. Carlstrom (Ed.), 2011 National Survey 
of Academic Advising (Monograph No. 25). Manhattan, KS: National Academic Advising 
Association. 


Rutschow, E., & Mayer, A. (2018). Early findings from a national survey of developmental 
education practices. New York, NY: Center for the Analysis of Postsecondary Readiness, 
Teachers College, Columbia University. 


Rutschow, E., & Schneider, E. (2011). Unlocking the gate: What we know about improving 
developmental education. New York, NY: MDRC. 


Rutschow, E., Sepanik, S., Deitch, V., Raufman, J., Dukes, D., & Moussa, A. (2019). Gaining 
ground: Findings from the Dana Center Mathematics Pathways Impact Study. New Y ork, 
NY: MDRC. 


65 


Schochet, P. Z. (2008). Technical methods report: Guidelines for multiple testing in impact 
evaluations (NCEE 2008-4018). Washington, DC: National Center for Education 
Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of 
Education. 


Scott-Clayton, J. (2012). Do high-stakes placement exams predict college success? (CCRC 
Working Paper No. 41). New York, NY: Community College Research Center, Teachers 
College, Columbia University. 


Scott-Clayton, J., & Schudde, L. (2017). Performance requirements in need-based aid: What 
roles do they serve, and do they work? [Research brief]. New York, NY: Center for 
Analysis of Postsecondary Education and Employment, Teachers College, Columbia 
University. 


Scrivener, S., Gupta, H., Weiss, M. J., Cohen, B., Cormier, M. S., & Brathwaite, J. (2018). 
Becoming college-ready: Early findings from a CUNY Start evaluation. New York, NY: 
MDRC. 


Scrivener, S., Weiss, M. J., & Sommo, C. (2012). What can a multifaceted program do for 
community college students? Early results from an evaluation of Accelerated Study in 
Associate Programs (ASAP) for developmental education students. New York, NY: 
MDRC. 


Sommo, C., Boynton, M., Collado, H., Diamond, J., Gardenhire, A., Ratledge, A., Rudd, T., 
Weiss, M. J. (2014). Mapping success: Performance-based scholarships, student services, 
and developmental math at Hillsborough Community College. New York, NY: MDRC. 


Stein, M. K., Smith, M. S., Henningsen, M. A., & Silver, E. A. (2009). Implementing standards- 
based mathematics instruction: A casebook for professional development. New York, NY: 
Teachers College Press. 


Stigler, J. W., Givvin, K. B., & Thompson, B. J. (2010). What community college 
developmental mathematics students understand about mathematics. MathAMATYC 
Educator, 1(3). 


Stigler, J. W., & Hiebert, J. (2009). The teaching gap: Best ideas from the world’s teachers for 
improving education in the classroom. New York, NY: Free Press. 


U.S. Department of Education. (2011). Building on foundations for success: Guidelines for 
improving adult mathematics instruction. Washington, DC: Office of Vocational and Adult 
Education. 


U.S. Department of Education. U.S. Department of Education, National Center for Education 
Statistics. (n.d.) QuickStats. https://nces.ed.gov/datalab/quickstats. Accessed on July 5, 
2018. 


Valentine, J. C., Konstantopoulos, S., & Goldrick-Rab, S. (2017). What happens to students 
placed into developmental education? A meta-analysis of regression discontinuity studies. 
Review of Educational Research, 87(4), 806-833. 


Visher, M. G., Wathington, H., Richburg-Hayes, L., & Schneider, E. (2008). The Learning 
Communities Demonstration: Rationale, Sites and Research Design. New York, NY: 
National Center for Postsecondary Research. 


66 


Visher, M. G., Weiss, M. J., Weissman, E., Rudd, T., & Wathington, H. D. (2012). The effects 
of learning communities for students in developmental education. New York, NY: MDRC. 


Weiss, M., Bloom, H., & Brock, T. (2014). A conceptual framework for studying the sources of 
variation in program effects. Journal of Policy Analysis and Management, 33(3), 778-808. 


Weiss, M., Brock, T., Sommo, C., Rudd, T., & Turner, M. C. (2011). Serving community 
college students on probation: Four-year findings from Chaffey College’s Opening Doors 
Program. New York, NY: MDRC. 


Weiss, M., Visher, M., Weissman, E., & Wathington, H. (2015). The impact of learning 
communities for students in developmental education: A synthesis of findings from 
randomized trials at six community colleges. Educational Evaluation and Policy Analysis, 
37(4), 520-541. 


Weiss, M. J., Bloom, H. S., Verbitsky-Savitz, N., Gupta, H., Vigil, A. E., & Cullinan, D. N. 
(2017). How much do the effects of education and training programs vary across sites? 
Evidence from past multisite randomized trials. Journal of Research on Educational 
Effectiveness, 10(4), 843-876. 


Weiss, M. J., Ratledge, A., Sommo, C., & Gupta, H. (2019). Supporting community college 
students from start to degree completion: Long-term evidence from a randomized trial of 
CUNY’s ASAP. American Economic Journal: Applied Economics, 11(3), 253-297. 


67 


